Oct 30 17:51:39.248860 ip-10-0-130-250 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:51:39.647954 ip-10-0-130-250 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:51:39.647954 ip-10-0-130-250 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:51:39.647954 ip-10-0-130-250 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:51:39.647954 ip-10-0-130-250 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:51:39.651695 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651612 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:39.651695 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651692 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:39.651695 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651696 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:39.651695 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651699 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:39.651695 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651702 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:39.651695 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651705 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651708 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651711 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651714 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651716 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651719 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651722 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651724 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651727 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651730 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651732 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651736 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651739 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651741 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651744 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651747 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651749 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651753 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651756 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651758 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:39.651911 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651760 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651763 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651766 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651775 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651778 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651781 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651783 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651786 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651789 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651794 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651799 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651803 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651806 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651808 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651812 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651815 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651818 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651820 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651823 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:39.652372 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651826 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651829 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651831 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651834 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651837 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651839 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651842 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651844 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651847 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651850 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651852 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651855 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651857 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651860 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651862 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651865 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651868 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651871 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651874 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651876 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:39.652837 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651879 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651882 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651884 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651887 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651889 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651892 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651895 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651898 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651900 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651904 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651907 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651909 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651912 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651915 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651918 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651920 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651924 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651926 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651929 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651931 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:39.653305 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651934 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651936 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651939 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651941 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651944 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651946 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651949 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651951 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651954 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651957 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651960 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651963 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651965 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651968 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651971 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651973 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651976 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651979 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651982 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:39.653798 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651985 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651989 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.651993 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654084 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654093 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654096 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654100 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654103 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654106 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654109 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654112 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654114 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654117 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654120 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654122 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654125 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654127 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654130 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654132 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:39.654249 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654137 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654140 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654143 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654145 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654148 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654151 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654154 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654157 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654159 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654162 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654165 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654168 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654170 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654173 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654175 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654179 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654182 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654185 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654187 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:39.654784 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654190 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654192 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654195 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654198 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654201 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654203 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654206 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654209 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654211 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654214 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654217 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654219 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654222 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654227 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654230 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654233 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654236 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654238 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654240 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:39.655253 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654244 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654246 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654249 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654252 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654255 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654257 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654260 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654264 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654268 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654271 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654274 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654277 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654279 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654282 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654285 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654288 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654290 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654293 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654296 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:39.655733 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654298 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654301 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654303 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654305 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654308 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654311 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654313 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654316 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654319 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654321 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654324 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654326 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654329 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654350 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654353 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654356 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654359 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654361 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654364 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654366 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:39.656190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654369 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654371 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654376 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654378 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654381 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654384 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654386 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654389 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654392 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654394 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654397 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654399 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654402 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654925 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654931 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654933 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654937 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654940 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654943 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654946 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:39.656687 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654948 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654951 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654954 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654956 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654959 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654961 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654965 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654969 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654972 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654975 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654978 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654980 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654983 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654986 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654989 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654993 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.654997 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655000 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655003 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:39.657158 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655006 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655008 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655011 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655014 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655016 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655019 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655021 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655024 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655026 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655029 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655031 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655033 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655036 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655038 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655041 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655044 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655046 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655049 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655052 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655054 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:39.657628 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655057 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655060 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655062 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655065 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655067 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655070 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655072 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655075 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655077 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655080 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655082 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655086 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655088 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655091 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655093 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655096 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655098 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655101 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655103 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:39.658093 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655106 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655108 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655111 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655113 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655116 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655119 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655121 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655124 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655126 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655129 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655131 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655134 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655136 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655139 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655142 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655144 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655147 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655149 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655151 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655155 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:39.658571 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655157 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655160 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655162 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655165 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655167 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655170 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655172 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655175 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655178 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655180 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655183 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655185 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655188 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655190 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655193 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655198 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655201 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655203 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655206 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655209 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:39.659059 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.655211 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656223 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656234 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656240 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656245 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656250 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656253 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656259 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656263 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656266 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656271 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656275 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656279 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656282 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656285 2578 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656288 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656291 2578 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656294 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656297 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656302 2578 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656306 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656309 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656312 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656316 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:51:39.659582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656320 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656323 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656326 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656331 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656335 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656338 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656341 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656344 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656347 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656351 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656354 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656357 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656360 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656363 2578 flags.go:64] FLAG: --enable-server="true" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656366 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656371 2578 flags.go:64] FLAG: --event-burst="100" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656374 2578 flags.go:64] FLAG: --event-qps="50" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656377 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656382 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656385 2578 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656390 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656393 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656396 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656399 2578 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656402 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:51:39.660152 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656406 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656409 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656412 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656415 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656418 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656421 2578 flags.go:64] FLAG: --feature-gates="" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656426 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656429 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656432 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656436 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656441 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656445 2578 flags.go:64] FLAG: --help="false" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656448 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-130-250.ec2.internal" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656451 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656454 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656457 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656460 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656464 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656467 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656470 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656473 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656476 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656479 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656482 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:51:39.660881 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656486 2578 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656491 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656494 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656497 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656500 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656503 2578 flags.go:64] FLAG: --lock-file="" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656506 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656509 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656512 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656517 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656533 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656549 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656552 2578 flags.go:64] FLAG: --logging-format="text" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656555 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656559 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656562 2578 flags.go:64] FLAG: --manifest-url="" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656565 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656570 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656573 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656579 2578 flags.go:64] FLAG: --max-pods="110" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656582 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656585 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656588 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656591 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656595 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:51:39.661466 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656598 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656601 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656608 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656612 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656614 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656618 2578 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656621 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656623 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656628 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656631 2578 flags.go:64] FLAG: --port="10250" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656635 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656638 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08935549a12d46347" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656641 2578 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656644 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656647 2578 flags.go:64] FLAG: --register-node="true" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656650 2578 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656654 2578 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656657 2578 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656660 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656663 2578 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656667 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656670 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656673 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656676 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656679 2578 flags.go:64] FLAG: --runonce="false" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656682 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:51:39.662098 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656686 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656689 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656692 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656695 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656699 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656703 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656706 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656709 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656712 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656715 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656718 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656721 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656724 2578 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656727 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656733 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656737 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656739 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656744 2578 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656747 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656750 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656753 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656755 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656758 2578 flags.go:64] FLAG: --v="2" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656767 2578 flags.go:64] FLAG: --version="false" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656771 2578 flags.go:64] FLAG: --vmodule="" Oct 30 17:51:39.662823 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656776 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.656779 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656920 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656923 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656926 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656934 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656937 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656940 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656943 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656945 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656948 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656950 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656953 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656955 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656958 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656961 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656963 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656966 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656968 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656971 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:39.663576 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656973 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656976 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656978 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656981 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656984 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656987 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656989 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656992 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656995 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656997 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.656999 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657002 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657005 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657007 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657010 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657012 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657014 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657017 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657019 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:39.664280 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657022 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657025 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657027 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657030 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657033 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657035 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657037 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657040 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657043 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657045 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657048 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657051 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657053 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657056 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657059 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657063 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657066 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657070 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657073 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:39.664765 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657076 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657079 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657081 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657084 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657086 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657089 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657091 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657094 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657096 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657100 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657105 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657108 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657111 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657114 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657116 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657119 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657121 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657124 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657126 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:39.665225 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657129 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657132 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657134 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657137 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657139 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657142 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657144 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657147 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657149 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657152 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657155 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657157 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657160 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657163 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657166 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657169 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657172 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657174 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657177 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657179 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:39.665767 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657182 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657184 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657187 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657189 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657192 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657194 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657197 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657199 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657202 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657204 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657207 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.657233 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657268 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657273 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657276 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:39.666238 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657280 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657283 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657286 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657290 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657294 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657298 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657303 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657307 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657310 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657312 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657316 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657319 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657321 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657324 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657327 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657329 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657332 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657335 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657338 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:39.666620 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657340 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657343 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657346 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657348 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657351 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657353 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657356 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657359 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657361 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657364 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657367 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657369 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657372 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657374 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657377 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657379 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657382 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657384 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657387 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657392 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:39.667089 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657395 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657398 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657400 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657418 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657422 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657426 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657429 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657432 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657435 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657437 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657440 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657443 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657445 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657448 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657450 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657453 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657456 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657459 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657461 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:39.667588 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657464 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657466 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657469 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657472 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657474 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657477 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657480 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657482 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657485 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657488 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657490 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657493 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657498 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657500 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657503 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657506 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657508 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657511 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657514 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657517 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:39.668022 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657534 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657537 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657540 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657543 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657546 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657548 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657551 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657553 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657556 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657559 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657561 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657564 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657566 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657569 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657571 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657574 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657576 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657579 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657583 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:39.668497 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657586 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657589 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657592 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657595 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657597 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.657602 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.665341 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.665359 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.665377 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.665382 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665444 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665449 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665452 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665455 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665457 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665460 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665463 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665466 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665468 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:39.668993 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665472 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665474 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665477 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665480 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665483 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665485 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665502 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665506 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665509 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665512 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665515 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665518 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665535 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665538 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665541 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665544 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665546 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665549 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665557 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665560 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:39.669447 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665563 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665566 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665568 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665571 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665574 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665577 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665579 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665582 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665585 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665587 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665589 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665592 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665595 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665597 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665600 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665602 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665605 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665608 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665612 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:39.670073 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665616 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665619 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665622 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665624 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665627 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665630 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665632 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665635 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665637 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665640 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665642 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665645 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665648 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665651 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665654 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665656 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665660 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665662 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665665 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665667 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:39.670514 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665670 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665673 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665675 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665677 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665680 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665683 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665685 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665687 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665690 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665693 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665695 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665698 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665701 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665704 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665706 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665709 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665711 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665713 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665716 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:39.671017 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665718 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665721 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665723 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665726 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665729 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665732 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665735 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665738 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665740 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665743 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665745 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665748 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665751 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665753 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665757 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665761 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665765 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665768 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:39.671463 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665771 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.665802 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665972 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665978 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665982 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665985 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665988 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665992 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665995 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.665998 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666002 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666004 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666007 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666010 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666012 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666015 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:39.671929 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666017 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666020 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666023 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666026 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666028 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666031 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666034 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666036 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666039 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666041 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666045 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666047 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666049 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666052 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666055 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666057 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666060 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666062 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666065 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:39.672308 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666067 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666070 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666072 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666074 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666077 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666081 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666083 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666086 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666088 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666091 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666093 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666096 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666099 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666101 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666104 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666107 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666110 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666112 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666115 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666118 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:39.672773 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666120 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666123 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666126 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666128 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666131 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666133 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666136 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666138 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666141 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666143 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666145 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666148 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666151 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666153 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666155 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666158 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666160 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666163 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666166 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666168 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:39.673307 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666171 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666173 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666176 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666179 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666181 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666184 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666188 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666192 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666195 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666198 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666200 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666203 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666207 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666211 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666214 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666218 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666221 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666224 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666228 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:39.673840 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666230 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666233 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666236 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666239 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666241 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666244 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666247 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666250 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666252 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666255 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666257 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666260 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666263 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666266 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.666289 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666322 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:51:39.674281 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666326 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666329 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666332 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666335 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666341 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666344 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666348 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666351 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666354 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666357 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666359 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666362 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666365 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666368 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666370 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666373 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666376 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666378 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666381 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:51:39.674704 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666383 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666386 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666389 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666391 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666394 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666396 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666399 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666401 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666404 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666406 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666409 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666413 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666415 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666417 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666420 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666423 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666425 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666431 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666433 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:51:39.675168 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666436 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666438 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666441 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666444 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666447 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666449 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666452 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666454 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666457 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666459 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666462 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666464 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666467 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666470 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666472 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666475 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666477 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666480 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666482 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666485 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:51:39.675680 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666487 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666489 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666493 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666495 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666499 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666501 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666504 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666507 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666509 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666512 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666517 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666536 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666541 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666544 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666547 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666549 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666552 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666555 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666557 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:51:39.676151 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666560 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666563 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666565 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666568 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666570 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666573 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666576 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666578 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666581 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666583 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666586 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666589 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666591 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666594 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666596 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666599 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666601 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666605 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666610 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:51:39.676619 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666613 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666617 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666621 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666625 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666629 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666632 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666634 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666637 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:39.666640 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.666704 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.668694 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.669746 2578 server.go:1014] "Starting client certificate rotation" Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.669836 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.669878 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.671784 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:51:39.677091 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.672926 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:51:39.683898 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.683879 2578 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:51:39.686992 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.686976 2578 log.go:25] "Validated CRI v1 image API" Oct 30 17:51:39.687891 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.687869 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:51:39.692632 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.692611 2578 fs.go:135] Filesystem UUIDs: map[1fb7ba10-e3a7-4048-b76f-ea246cdc9860:/dev/nvme0n1p4 2288559e-90c2-49ee-98cf-0e4c6ee82ca7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 17:51:39.692685 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.692632 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 17:51:39.698351 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.698218 2578 manager.go:217] Machine: {Timestamp:2026-10-30 17:51:39.697053859 +0000 UTC m=+0.331159659 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097255 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2640a9e10de29d989d3321f2edf96b SystemUUID:ec2640a9-e10d-e29d-989d-3321f2edf96b BootID:80b88d9f-8980-4d85-a6e8-b7efa1c25c39 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:66:b1:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:66:b1:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:5b:5e:33:2c:05 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:51:39.698820 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.698805 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:51:39.698949 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.698932 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:51:39.699982 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.699950 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:51:39.700147 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.699983 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-250.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:51:39.700233 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.700152 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:51:39.700233 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.700163 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:51:39.700233 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.700213 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:51:39.700328 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.700236 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:51:39.701022 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.701008 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:51:39.703603 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.703588 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:51:39.703738 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.703726 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:51:39.704506 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.704495 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:51:39.704573 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.704513 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:51:39.704573 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.704540 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:51:39.704573 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.704554 2578 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:51:39.704573 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.704567 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:51:39.704925 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.704913 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:51:39.707132 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.707110 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:51:39.711656 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.711642 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:51:39.712534 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.712513 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:51:39.712575 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.712556 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:51:39.712715 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.712705 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:51:39.712743 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.712721 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:51:39.712743 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.712727 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:51:39.712743 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.712732 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:51:39.712743 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.712738 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:51:39.712743 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.712744 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:51:39.713283 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.713275 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:51:39.713311 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.713285 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:51:39.713311 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.713292 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:51:39.713311 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.713298 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:51:39.713311 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.713307 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:51:39.713430 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.713315 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:51:39.713758 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.713749 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:51:39.713758 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.713758 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:51:39.717332 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.717320 2578 server.go:1263] "Started kubelet" Oct 30 17:51:39.717474 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.717407 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:51:39.717513 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.717449 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:51:39.717513 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.717492 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:51:39.718145 ip-10-0-130-250 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:51:39.718687 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.718667 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:51:39.720326 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.720309 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:51:39.724376 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.724352 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 17:51:39.728031 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.728013 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:51:39.728486 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.728460 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:51:39.728704 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.728662 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:39.729074 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.729055 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:51:39.729160 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.729085 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:51:39.729160 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.729058 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:51:39.729160 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.729153 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:51:39.729160 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.729158 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:51:39.729508 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.729488 2578 factory.go:153] Registering CRI-O factory Oct 30 17:51:39.729508 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.729509 2578 factory.go:223] Registration of the crio container factory successfully Oct 30 17:51:39.729672 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.729587 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:51:39.729672 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.729597 2578 factory.go:55] Registering systemd factory Oct 30 17:51:39.729672 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.729605 2578 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:51:39.729672 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.729623 2578 factory.go:103] Registering Raw factory Oct 30 17:51:39.729672 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.729638 2578 manager.go:1199] Started watching for new ooms in manager Oct 30 17:51:39.730575 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.730561 2578 manager.go:319] Starting recovery of all containers Oct 30 17:51:39.744553 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.744461 2578 manager.go:324] Recovery completed Oct 30 17:51:39.749621 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.749605 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:39.751576 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.751550 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:39.751666 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.751588 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:39.751666 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.751598 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:39.752604 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.752588 2578 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:51:39.752698 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.752680 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:51:39.752796 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.752708 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:51:39.752796 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.752590 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:39.754014 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.753999 2578 policy_none.go:50] "Start" Oct 30 17:51:39.754014 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.754015 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:51:39.754124 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.754032 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:51:39.754337 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.754322 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:39.754406 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.754353 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:39.754406 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.754368 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:39.755202 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.755187 2578 policy_none.go:44] "Start" Oct 30 17:51:39.755360 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.755336 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-250.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:51:39.755435 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.755335 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 17:51:39.756596 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.755594 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-250.ec2.internal.18e360104afaf459 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-250.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-250.ec2.internal,},FirstTimestamp:2026-10-30 17:51:39.717293145 +0000 UTC m=+0.351398944,LastTimestamp:2026-10-30 17:51:39.717293145 +0000 UTC m=+0.351398944,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-250.ec2.internal,}" Oct 30 17:51:39.776906 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.776837 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-250.ec2.internal.18e360104d060bda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-250.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-250.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-250.ec2.internal,},FirstTimestamp:2026-10-30 17:51:39.75157449 +0000 UTC m=+0.385680293,LastTimestamp:2026-10-30 17:51:39.75157449 +0000 UTC m=+0.385680293,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-250.ec2.internal,}" Oct 30 17:51:39.777012 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.776937 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:51:39.777012 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.776954 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:51:39.777108 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.777092 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:51:39.781156 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.781137 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:51:39.795300 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.795271 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:51:39.798595 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.798581 2578 manager.go:341] "Starting Device Plugin manager" Oct 30 17:51:39.798710 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.798607 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:51:39.798710 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.798615 2578 server.go:87] "Starting device plugin registration server" Oct 30 17:51:39.798903 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.798889 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:51:39.798963 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.798904 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:51:39.799125 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.799111 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:51:39.799192 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.799183 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:51:39.799245 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.799193 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:51:39.799842 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.799824 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:51:39.799904 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.799872 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:39.803964 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.803884 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-250.ec2.internal.18e360104d065434 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-250.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-250.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-250.ec2.internal,},FirstTimestamp:2026-10-30 17:51:39.751593012 +0000 UTC m=+0.385698813,LastTimestamp:2026-10-30 17:51:39.751593012 +0000 UTC m=+0.385698813,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-250.ec2.internal,}" Oct 30 17:51:39.819472 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.810120 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z5xgv" Oct 30 17:51:39.820588 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.820048 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z5xgv" Oct 30 17:51:39.832670 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.832589 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-250.ec2.internal.18e360104d06763e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-250.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-250.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-250.ec2.internal,},FirstTimestamp:2026-10-30 17:51:39.751601726 +0000 UTC m=+0.385707526,LastTimestamp:2026-10-30 17:51:39.751601726 +0000 UTC m=+0.385707526,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-250.ec2.internal,}" Oct 30 17:51:39.854216 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.854173 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:51:39.855404 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.855384 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:51:39.855493 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.855413 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:51:39.855493 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.855437 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:51:39.855493 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.855477 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 17:51:39.871271 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.871246 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:39.899667 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.899601 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:39.900555 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.900515 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:39.900636 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.900565 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:39.900636 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.900575 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:39.900636 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.900600 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-250.ec2.internal" Oct 30 17:51:39.921801 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.921634 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-250.ec2.internal" Oct 30 17:51:39.921907 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.921805 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-250.ec2.internal\": node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:39.956012 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.955982 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal"] Oct 30 17:51:39.956123 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.956051 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:39.956938 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.956923 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:39.957038 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.956962 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:39.957038 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.956972 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:39.958249 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.958237 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:39.958416 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.958401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:51:39.958463 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.958431 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:39.958957 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.958939 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:39.959030 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.958967 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:39.959030 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.958977 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:39.959030 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.959010 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:39.959030 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.959028 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:39.959157 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.959037 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:39.960037 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.960024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:51:39.960084 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.960047 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:39.960753 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.960738 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:39.960850 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.960796 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:39.960850 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.960815 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:39.960850 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:39.960827 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:39.995439 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.995408 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-250.ec2.internal\" not found" node="ip-10-0-130-250.ec2.internal" Oct 30 17:51:39.999083 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:39.999064 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-250.ec2.internal\" not found" node="ip-10-0-130-250.ec2.internal" Oct 30 17:51:40.030709 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:40.030679 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/49b07006ee51f49de29dbabbdd01d907-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" (UID: \"49b07006ee51f49de29dbabbdd01d907\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:51:40.030709 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:40.030710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49b07006ee51f49de29dbabbdd01d907-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" (UID: \"49b07006ee51f49de29dbabbdd01d907\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:51:40.030879 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:40.030732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef96f59234cd19c8b7af6a47596c7964-config\") pod \"kube-apiserver-proxy-ip-10-0-130-250.ec2.internal\" (UID: \"ef96f59234cd19c8b7af6a47596c7964\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:51:40.061100 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:40.061075 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:40.131790 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:40.131759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/49b07006ee51f49de29dbabbdd01d907-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" (UID: \"49b07006ee51f49de29dbabbdd01d907\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:51:40.131790 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:40.131791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49b07006ee51f49de29dbabbdd01d907-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" (UID: \"49b07006ee51f49de29dbabbdd01d907\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:51:40.132001 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:40.131807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef96f59234cd19c8b7af6a47596c7964-config\") pod \"kube-apiserver-proxy-ip-10-0-130-250.ec2.internal\" (UID: \"ef96f59234cd19c8b7af6a47596c7964\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:51:40.132001 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:40.131849 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/49b07006ee51f49de29dbabbdd01d907-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" (UID: \"49b07006ee51f49de29dbabbdd01d907\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:51:40.132001 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:40.131864 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef96f59234cd19c8b7af6a47596c7964-config\") pod \"kube-apiserver-proxy-ip-10-0-130-250.ec2.internal\" (UID: \"ef96f59234cd19c8b7af6a47596c7964\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:51:40.132001 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:40.131875 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49b07006ee51f49de29dbabbdd01d907-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" (UID: \"49b07006ee51f49de29dbabbdd01d907\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:51:40.161540 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:40.161439 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:40.261765 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:40.261733 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:40.298266 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:40.298230 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:51:40.302403 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:40.302088 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:51:40.362173 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:40.362135 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:41.489022 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:41.489000 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:41.589488 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:41.589456 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:41.690054 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:41.690024 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:41.696191 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:41.696170 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 17:51:41.696361 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:41.696341 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 17:51:41.790660 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:41.790631 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:41.841206 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:41.841170 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:51:41.842584 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:41.842561 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:51:41.842696 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:41.842599 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:51:41.842696 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:41.842615 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:51:41.849846 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:41.849817 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:46:40 +0000 UTC" deadline="2028-06-16 18:06:47.064833572 +0000 UTC" Oct 30 17:51:41.849846 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:41.849844 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14280h15m5.214992519s" Oct 30 17:51:41.881474 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:41.881448 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef96f59234cd19c8b7af6a47596c7964.slice/crio-cfd8e5655731d2860f1ddab8ce6a6ddb225411b8348500bbf21f6e92f55f1903 WatchSource:0}: Error finding container cfd8e5655731d2860f1ddab8ce6a6ddb225411b8348500bbf21f6e92f55f1903: Status 404 returned error can't find the container with id cfd8e5655731d2860f1ddab8ce6a6ddb225411b8348500bbf21f6e92f55f1903 Oct 30 17:51:41.882379 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:41.882357 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b07006ee51f49de29dbabbdd01d907.slice/crio-c3f27e6dc43e7301384b86d9bda5ce54b9cea35fe48942ba6912fffa683f275b WatchSource:0}: Error finding container c3f27e6dc43e7301384b86d9bda5ce54b9cea35fe48942ba6912fffa683f275b: Status 404 returned error can't find the container with id c3f27e6dc43e7301384b86d9bda5ce54b9cea35fe48942ba6912fffa683f275b Oct 30 17:51:41.887695 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:41.887680 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:51:41.890765 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:41.890751 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:41.991336 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:41.991288 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:51:42.032876 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.032824 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:42.055940 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.055914 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:51:42.081135 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.081111 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:51:42.083052 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.083038 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:51:42.105547 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.105525 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:51:42.165350 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.165326 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:42.566226 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.566196 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:42.732520 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.732491 2578 apiserver.go:52] "Watching apiserver" Oct 30 17:51:42.742128 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.742102 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:51:42.755087 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.755060 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 17:51:42.763729 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.763708 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:51:42.764064 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.764047 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal","openshift-multus/multus-additional-cni-plugins-2sfml","openshift-dns/node-resolver-xbxvk","openshift-multus/multus-md4zx","openshift-multus/network-metrics-daemon-dl2lm","openshift-ovn-kubernetes/ovnkube-node-k6zpc","kube-system/konnectivity-agent-vdlbp","kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal","openshift-network-diagnostics/network-check-target-vqltd","openshift-network-operator/iptables-alerter-wbzt5","kube-system/global-pull-secret-syncer-fglxs","openshift-image-registry/node-ca-7gt8q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5","openshift-cluster-node-tuning-operator/tuned-dpch6"] Oct 30 17:51:42.766490 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.766466 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:51:42.768701 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.768676 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:51:42.770913 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.770895 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.773194 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.773171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:42.773334 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:42.773249 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:51:42.775542 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.775520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.777950 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.777915 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:51:42.780711 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.780692 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:42.780800 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:42.780769 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:51:42.783634 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.783552 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:51:42.783737 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.783719 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.783796 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.783777 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5kfdv\"" Oct 30 17:51:42.783846 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.783831 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:51:42.787805 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.787782 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:42.787905 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:42.787867 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:51:42.788700 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.788680 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:51:42.788840 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.788737 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:51:42.789589 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.789541 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:51:42.790265 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.789747 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hhs6k\"" Oct 30 17:51:42.791445 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.790782 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:51:42.791445 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.790975 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:51:42.791445 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.791283 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:51:42.791445 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.791347 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dqbz4\"" Oct 30 17:51:42.791838 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.791823 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:51:42.792052 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.792029 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:51:42.792810 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.792791 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.793080 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.793046 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:51:42.795848 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.795828 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.810492 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.810468 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:51:42.810586 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.810539 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-glcn9" Oct 30 17:51:42.817824 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.817766 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-glcn9" Oct 30 17:51:42.856857 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.856834 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:51:42.868420 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868397 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-system-cni-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.868558 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868432 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07abda71-7d42-43e6-9878-90ab7da7d9c4-cni-binary-copy\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.868558 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868450 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-conf-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.868558 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868466 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cnibin\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.868558 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-cni-netd\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.868558 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.868804 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.868804 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868622 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.868804 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868647 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4c590f5-af8b-4994-8b03-7f5cee045767-ovn-node-metrics-cert\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.868804 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-lib-modules\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.868804 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-hostroot\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.868804 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-dbus\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:42.868804 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtx8j\" (UniqueName: \"kubernetes.io/projected/19bd958f-7d20-4da4-818f-06e23e01ab37-kube-api-access-rtx8j\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:42.868804 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868780 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-env-overrides\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.869130 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-ovnkube-script-lib\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.869130 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-registration-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.869130 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-modprobe-d\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.869130 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-multus-certs\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.869130 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.868971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s2j6\" (UniqueName: \"kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6\") pod \"network-check-target-vqltd\" (UID: \"d5d1adae-1728-499a-9d16-ec24f1d69160\") " pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:42.869130 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-slash\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.869130 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-cni-bin\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.869130 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-systemd\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.869479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-hosts-file\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:51:42.869479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-cnibin\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.869479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-system-cni-dir\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.869479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-serviceca\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:51:42.869479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869311 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-var-lib-kubelet\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.869479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869345 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmnr8\" (UniqueName: \"kubernetes.io/projected/5e94fb26-859a-48d2-a493-200543aa1881-kube-api-access-vmnr8\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.869479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhvw\" (UniqueName: \"kubernetes.io/projected/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-kube-api-access-plhvw\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:51:42.869479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-socket-dir-parent\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.869479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-k8s-cni-cncf-io\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.869479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869475 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-cni-bin\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869502 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-host\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysctl-d\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-netns\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-ovnkube-config\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869648 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-sys\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869713 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e94fb26-859a-48d2-a493-200543aa1881-tmp\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-etc-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692s4\" (UniqueName: \"kubernetes.io/projected/d4c590f5-af8b-4994-8b03-7f5cee045767-kube-api-access-692s4\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ce04018-ba31-464c-8996-967b9100f174-agent-certs\") pod \"konnectivity-agent-vdlbp\" (UID: \"2ce04018-ba31-464c-8996-967b9100f174\") " pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869817 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6kzk\" (UniqueName: \"kubernetes.io/projected/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-kube-api-access-r6kzk\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869843 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-kubelet-config\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-kubelet\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.869984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869913 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-run-netns\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.869985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6mwh\" (UniqueName: \"kubernetes.io/projected/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-kube-api-access-j6mwh\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-run\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-host-slash\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-cni-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870265 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-systemd-units\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-run-ovn-kubernetes\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-daemon-config\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ce04018-ba31-464c-8996-967b9100f174-konnectivity-ca\") pod \"konnectivity-agent-vdlbp\" (UID: \"2ce04018-ba31-464c-8996-967b9100f174\") " pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-socket-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-device-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-node-log\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.870714 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870435 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysconfig\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysctl-conf\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-tmp-dir\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-iptables-alerter-script\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h54zm\" (UniqueName: \"kubernetes.io/projected/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-kube-api-access-h54zm\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870559 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-kubelet\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-kubernetes\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870610 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-etc-kubernetes\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870647 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlkgn\" (UniqueName: \"kubernetes.io/projected/07abda71-7d42-43e6-9878-90ab7da7d9c4-kube-api-access-xlkgn\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-systemd\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-var-lib-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870724 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-ovn\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870748 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-sys-fs\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-host\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e94fb26-859a-48d2-a493-200543aa1881-etc-tuned\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-cni-multus\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870852 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-log-socket\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.871482 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.870876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgj9b\" (UniqueName: \"kubernetes.io/projected/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-kube-api-access-mgj9b\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:51:42.883511 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.883483 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:51:42.883688 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.883668 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:51:42.883759 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.883705 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:51:42.883816 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.883755 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:51:42.883816 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.883768 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:51:42.883911 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.883676 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:51:42.883911 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.883882 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:51:42.884036 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.883994 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:51:42.884036 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.883995 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:51:42.884036 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884011 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:51:42.884195 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884088 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:51:42.884195 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884135 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:51:42.884195 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884177 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nb8hn\"" Oct 30 17:51:42.884339 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884197 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:51:42.884339 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884221 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lsd9l\"" Oct 30 17:51:42.884339 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884232 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:51:42.884339 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884263 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k56gt\"" Oct 30 17:51:42.884339 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884268 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fzft4\"" Oct 30 17:51:42.884339 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884298 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:51:42.884339 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884309 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:51:42.884339 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884319 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:51:42.884678 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884350 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-twdt2\"" Oct 30 17:51:42.884678 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.884361 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k5mms\"" Oct 30 17:51:42.886445 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.886397 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" event={"ID":"ef96f59234cd19c8b7af6a47596c7964","Type":"ContainerStarted","Data":"cfd8e5655731d2860f1ddab8ce6a6ddb225411b8348500bbf21f6e92f55f1903"} Oct 30 17:51:42.887381 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.887355 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" event={"ID":"49b07006ee51f49de29dbabbdd01d907","Type":"ContainerStarted","Data":"c3f27e6dc43e7301384b86d9bda5ce54b9cea35fe48942ba6912fffa683f275b"} Oct 30 17:51:42.971830 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.971794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.972029 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.971840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.972094 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972077 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.972148 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-host\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:51:42.972201 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysctl-d\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.972201 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-netns\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.972301 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-ovnkube-config\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.972301 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-sys\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.972301 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e94fb26-859a-48d2-a493-200543aa1881-tmp\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.972301 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972281 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-etc-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.972479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692s4\" (UniqueName: \"kubernetes.io/projected/d4c590f5-af8b-4994-8b03-7f5cee045767-kube-api-access-692s4\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.972479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ce04018-ba31-464c-8996-967b9100f174-agent-certs\") pod \"konnectivity-agent-vdlbp\" (UID: \"2ce04018-ba31-464c-8996-967b9100f174\") " pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:51:42.972479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6kzk\" (UniqueName: \"kubernetes.io/projected/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-kube-api-access-r6kzk\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.972479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-kubelet-config\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:42.972479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972396 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.972479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:42.972479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-kubelet\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.972479 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-run-netns\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.972809 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.972809 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6mwh\" (UniqueName: \"kubernetes.io/projected/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-kube-api-access-j6mwh\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.972809 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-run\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.972960 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-host-slash\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:51:42.972960 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972901 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-netns\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.972960 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:42.972927 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:42.973099 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-cni-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.973099 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.973099 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.972992 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.973099 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.973099 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973010 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:51:42.973099 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:42.973099 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:42.973100 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs podName:19bd958f-7d20-4da4-818f-06e23e01ab37 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:43.473076452 +0000 UTC m=+3.080770875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs") pod "network-metrics-daemon-dl2lm" (UID: "19bd958f-7d20-4da4-818f-06e23e01ab37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:42.973417 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-systemd-units\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.973417 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:42.973160 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:42.973417 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-run-ovn-kubernetes\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.973417 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-daemon-config\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.973417 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:42.973221 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret podName:44ce8b1c-6ab2-4f83-a1f6-183cc6cea104 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:43.473207802 +0000 UTC m=+3.080902205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret") pod "global-pull-secret-syncer-fglxs" (UID: "44ce8b1c-6ab2-4f83-a1f6-183cc6cea104") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:42.973417 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ce04018-ba31-464c-8996-967b9100f174-konnectivity-ca\") pod \"konnectivity-agent-vdlbp\" (UID: \"2ce04018-ba31-464c-8996-967b9100f174\") " pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:51:42.973417 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-socket-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.973417 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-device-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.973417 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-node-log\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.973807 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973460 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-node-log\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.973807 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973521 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-ovnkube-config\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.973807 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973587 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-host\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:51:42.973807 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973650 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-cni-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.973807 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973687 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysctl-d\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.973807 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-daemon-config\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.974088 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973866 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-kubelet-config\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:42.974088 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.974071 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-etc-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.974248 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.974212 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-systemd-units\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.974248 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.974209 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.974355 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.974307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-run-ovn-kubernetes\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.974507 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.974483 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-socket-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.974573 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.974558 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-device-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.974804 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.974760 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-run\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.974878 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.973036 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-kubelet\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.974997 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.974916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysconfig\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.975049 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975017 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-run-netns\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.975049 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-sys\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.975049 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysconfig\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.975171 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysctl-conf\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.975171 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-tmp-dir\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:51:42.975171 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-iptables-alerter-script\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:51:42.975297 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h54zm\" (UniqueName: \"kubernetes.io/projected/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-kube-api-access-h54zm\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:51:42.975297 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975187 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-host-slash\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:51:42.975297 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-kubelet\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.975428 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-kubernetes\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.975428 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-etc-kubernetes\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.975428 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlkgn\" (UniqueName: \"kubernetes.io/projected/07abda71-7d42-43e6-9878-90ab7da7d9c4-kube-api-access-xlkgn\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.975428 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-systemd\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.975594 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-var-lib-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.975594 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-ovn\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.975594 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-sys-fs\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.975594 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-host\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.975594 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975537 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e94fb26-859a-48d2-a493-200543aa1881-etc-tuned\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.975594 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-cni-multus\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.975844 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-log-socket\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.975844 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975730 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-iptables-alerter-script\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:51:42.975844 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-kubelet\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.975844 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975838 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-kubernetes\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.976049 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975885 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-etc-kubernetes\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.976260 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976235 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-systemd\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.976260 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.975731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgj9b\" (UniqueName: \"kubernetes.io/projected/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-kube-api-access-mgj9b\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:51:42.976372 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976281 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-system-cni-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.976372 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07abda71-7d42-43e6-9878-90ab7da7d9c4-cni-binary-copy\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.976372 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-conf-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.976372 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976356 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysctl-conf\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.976546 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cnibin\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.976546 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-cni-netd\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.976546 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.976546 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.976546 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.976759 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4c590f5-af8b-4994-8b03-7f5cee045767-ovn-node-metrics-cert\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.976759 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-lib-modules\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.976759 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-hostroot\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.976759 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-dbus\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:42.976759 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976695 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-tmp-dir\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:51:42.976759 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtx8j\" (UniqueName: \"kubernetes.io/projected/19bd958f-7d20-4da4-818f-06e23e01ab37-kube-api-access-rtx8j\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:42.977033 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-env-overrides\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.977033 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-ovnkube-script-lib\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.977033 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-registration-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.977033 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-modprobe-d\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.977033 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-multus-certs\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.977033 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.976922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2j6\" (UniqueName: \"kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6\") pod \"network-check-target-vqltd\" (UID: \"d5d1adae-1728-499a-9d16-ec24f1d69160\") " pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:42.977033 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977011 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ce04018-ba31-464c-8996-967b9100f174-konnectivity-ca\") pod \"konnectivity-agent-vdlbp\" (UID: \"2ce04018-ba31-464c-8996-967b9100f174\") " pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:51:42.977033 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-slash\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.977375 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-cni-bin\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.977375 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-systemd\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.977375 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977199 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.977375 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-hosts-file\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:51:42.977375 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977204 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e94fb26-859a-48d2-a493-200543aa1881-tmp\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.977375 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-cnibin\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.977375 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977273 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-ovn\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.977375 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-system-cni-dir\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.977375 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977310 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.977375 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-serviceca\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:51:42.977375 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-var-lib-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.977375 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-var-lib-kubelet\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-conf-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-system-cni-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmnr8\" (UniqueName: \"kubernetes.io/projected/5e94fb26-859a-48d2-a493-200543aa1881-kube-api-access-vmnr8\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977448 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-cni-netd\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhvw\" (UniqueName: \"kubernetes.io/projected/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-kube-api-access-plhvw\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-log-socket\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977549 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-socket-dir-parent\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977621 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-var-lib-kubelet\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-k8s-cni-cncf-io\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-k8s-cni-cncf-io\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-cni-bin\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977723 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-cni-multus\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977767 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-cni-bin\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977798 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-hosts-file\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977828 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-systemd\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.977887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977837 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-modprobe-d\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977892 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-sys-fs\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977895 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-slash\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977927 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-system-cni-dir\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977967 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-registration-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.977984 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-dbus\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.978016 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-host\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.978062 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-hostroot\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.978101 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cnibin\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.978363 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ce04018-ba31-464c-8996-967b9100f174-agent-certs\") pod \"konnectivity-agent-vdlbp\" (UID: \"2ce04018-ba31-464c-8996-967b9100f174\") " pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.978369 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-env-overrides\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.978518 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-multus-certs\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.978513 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-cnibin\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.978677 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.978633 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-cni-bin\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.979248 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.978823 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-serviceca\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:51:42.979248 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.978884 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-ovnkube-script-lib\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.979248 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.978985 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-lib-modules\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.979248 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.979105 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-socket-dir-parent\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.979801 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.979778 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07abda71-7d42-43e6-9878-90ab7da7d9c4-cni-binary-copy\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:42.981869 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.981846 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e94fb26-859a-48d2-a493-200543aa1881-etc-tuned\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:42.981996 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.981900 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4c590f5-af8b-4994-8b03-7f5cee045767-ovn-node-metrics-cert\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.999859 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.999760 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6mwh\" (UniqueName: \"kubernetes.io/projected/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-kube-api-access-j6mwh\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:42.999995 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.999958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-692s4\" (UniqueName: \"kubernetes.io/projected/d4c590f5-af8b-4994-8b03-7f5cee045767-kube-api-access-692s4\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:42.999995 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:42.999967 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6kzk\" (UniqueName: \"kubernetes.io/projected/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-kube-api-access-r6kzk\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:43.001879 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.001836 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlkgn\" (UniqueName: \"kubernetes.io/projected/07abda71-7d42-43e6-9878-90ab7da7d9c4-kube-api-access-xlkgn\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:51:43.002398 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.002346 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h54zm\" (UniqueName: \"kubernetes.io/projected/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-kube-api-access-h54zm\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:51:43.003260 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.003237 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtx8j\" (UniqueName: \"kubernetes.io/projected/19bd958f-7d20-4da4-818f-06e23e01ab37-kube-api-access-rtx8j\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:43.003889 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.003844 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmnr8\" (UniqueName: \"kubernetes.io/projected/5e94fb26-859a-48d2-a493-200543aa1881-kube-api-access-vmnr8\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:43.004370 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.004323 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhvw\" (UniqueName: \"kubernetes.io/projected/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-kube-api-access-plhvw\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:51:43.005405 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.005240 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:43.005405 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.005262 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:43.005405 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.005277 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4s2j6 for pod openshift-network-diagnostics/network-check-target-vqltd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:43.005405 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.005362 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6 podName:d5d1adae-1728-499a-9d16-ec24f1d69160 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:43.505342657 +0000 UTC m=+3.113037075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4s2j6" (UniqueName: "kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6") pod "network-check-target-vqltd" (UID: "d5d1adae-1728-499a-9d16-ec24f1d69160") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:43.005665 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.005412 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgj9b\" (UniqueName: \"kubernetes.io/projected/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-kube-api-access-mgj9b\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:51:43.082886 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.082801 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:51:43.093141 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.093108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-md4zx" Oct 30 17:51:43.095322 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:43.095293 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c8fb7c_e9a6_4eff_9a66_b70848c03a95.slice/crio-9f8dee39d1523e2e2efed08471bf46398d73d72028d5b0edae754cac4eb95746 WatchSource:0}: Error finding container 9f8dee39d1523e2e2efed08471bf46398d73d72028d5b0edae754cac4eb95746: Status 404 returned error can't find the container with id 9f8dee39d1523e2e2efed08471bf46398d73d72028d5b0edae754cac4eb95746 Oct 30 17:51:43.102196 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.102171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:51:43.103330 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:43.103299 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07abda71_7d42_43e6_9878_90ab7da7d9c4.slice/crio-3952840a381d0ce2f4744ec7477a9cf0c6c5fae507720774505f073f2d2e8bbb WatchSource:0}: Error finding container 3952840a381d0ce2f4744ec7477a9cf0c6c5fae507720774505f073f2d2e8bbb: Status 404 returned error can't find the container with id 3952840a381d0ce2f4744ec7477a9cf0c6c5fae507720774505f073f2d2e8bbb Oct 30 17:51:43.112632 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:43.112600 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e25bb5_1c5b_449e_a7f0_0deb010035ed.slice/crio-19cd058eb83329fa6bc4277bf0e5e8faa72e400ef58a8faa61470589005876c9 WatchSource:0}: Error finding container 19cd058eb83329fa6bc4277bf0e5e8faa72e400ef58a8faa61470589005876c9: Status 404 returned error can't find the container with id 19cd058eb83329fa6bc4277bf0e5e8faa72e400ef58a8faa61470589005876c9 Oct 30 17:51:43.121118 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.121097 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:51:43.132121 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:43.132093 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce04018_ba31_464c_8996_967b9100f174.slice/crio-3866e6ffabee47730af52b48828cedea8224c425c2f6bcbf19d40c024946200e WatchSource:0}: Error finding container 3866e6ffabee47730af52b48828cedea8224c425c2f6bcbf19d40c024946200e: Status 404 returned error can't find the container with id 3866e6ffabee47730af52b48828cedea8224c425c2f6bcbf19d40c024946200e Oct 30 17:51:43.135490 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.135469 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:51:43.137516 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.137494 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:51:43.147850 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:43.147822 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c590f5_af8b_4994_8b03_7f5cee045767.slice/crio-0cd0faf02ac1d0545ae1275e67b860caa84ccb64b1c0455eab979dbaad30084b WatchSource:0}: Error finding container 0cd0faf02ac1d0545ae1275e67b860caa84ccb64b1c0455eab979dbaad30084b: Status 404 returned error can't find the container with id 0cd0faf02ac1d0545ae1275e67b860caa84ccb64b1c0455eab979dbaad30084b Oct 30 17:51:43.151143 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.148608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:51:43.152076 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.152011 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:51:43.154852 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.154828 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:51:43.160428 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:43.160390 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e94fb26_859a_48d2_a493_200543aa1881.slice/crio-2a01bbb92e6b369de1601a8484e5b68b7793b750c5aa9c7ba8310ef75a49c372 WatchSource:0}: Error finding container 2a01bbb92e6b369de1601a8484e5b68b7793b750c5aa9c7ba8310ef75a49c372: Status 404 returned error can't find the container with id 2a01bbb92e6b369de1601a8484e5b68b7793b750c5aa9c7ba8310ef75a49c372 Oct 30 17:51:43.166099 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:43.166077 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6f83f6_600c_4fa7_a20b_6a179c69bffc.slice/crio-e9f90e7f00f9665d52b7a1c5f5b916dbc31411be8d913d761cf750fc3e6f3ede WatchSource:0}: Error finding container e9f90e7f00f9665d52b7a1c5f5b916dbc31411be8d913d761cf750fc3e6f3ede: Status 404 returned error can't find the container with id e9f90e7f00f9665d52b7a1c5f5b916dbc31411be8d913d761cf750fc3e6f3ede Oct 30 17:51:43.166637 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:51:43.166604 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4a5fdf_7fb3_4e25_9dc7_7f48c6c06881.slice/crio-c8f683348012bd6873a25ecaf7c515e4200ff75713d9f8105822f8efbec1d83f WatchSource:0}: Error finding container c8f683348012bd6873a25ecaf7c515e4200ff75713d9f8105822f8efbec1d83f: Status 404 returned error can't find the container with id c8f683348012bd6873a25ecaf7c515e4200ff75713d9f8105822f8efbec1d83f Oct 30 17:51:43.482088 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.481741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:43.482088 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.481799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:43.482088 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.481985 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:43.482088 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.482046 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret podName:44ce8b1c-6ab2-4f83-a1f6-183cc6cea104 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:44.482027616 +0000 UTC m=+4.089722039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret") pod "global-pull-secret-syncer-fglxs" (UID: "44ce8b1c-6ab2-4f83-a1f6-183cc6cea104") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:43.482347 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.482141 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:43.482347 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.482200 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs podName:19bd958f-7d20-4da4-818f-06e23e01ab37 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:44.482183159 +0000 UTC m=+4.089877574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs") pod "network-metrics-daemon-dl2lm" (UID: "19bd958f-7d20-4da4-818f-06e23e01ab37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:43.582925 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.582700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2j6\" (UniqueName: \"kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6\") pod \"network-check-target-vqltd\" (UID: \"d5d1adae-1728-499a-9d16-ec24f1d69160\") " pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:43.583122 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.583107 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:43.583195 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.583170 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:43.583195 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.583185 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4s2j6 for pod openshift-network-diagnostics/network-check-target-vqltd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:43.583292 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.583259 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6 podName:d5d1adae-1728-499a-9d16-ec24f1d69160 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:44.583241904 +0000 UTC m=+4.190936319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4s2j6" (UniqueName: "kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6") pod "network-check-target-vqltd" (UID: "d5d1adae-1728-499a-9d16-ec24f1d69160") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:43.822166 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.821971 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:46:42 +0000 UTC" deadline="2028-06-23 10:32:52.321996376 +0000 UTC" Oct 30 17:51:43.822166 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.822031 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14440h41m8.499968696s" Oct 30 17:51:43.882175 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.882140 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:43.882355 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:43.882270 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:51:43.899793 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.899749 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" event={"ID":"9d6f83f6-600c-4fa7-a20b-6a179c69bffc","Type":"ContainerStarted","Data":"e9f90e7f00f9665d52b7a1c5f5b916dbc31411be8d913d761cf750fc3e6f3ede"} Oct 30 17:51:43.907784 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.907750 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dpch6" event={"ID":"5e94fb26-859a-48d2-a493-200543aa1881","Type":"ContainerStarted","Data":"2a01bbb92e6b369de1601a8484e5b68b7793b750c5aa9c7ba8310ef75a49c372"} Oct 30 17:51:43.909669 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.909625 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7gt8q" event={"ID":"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0","Type":"ContainerStarted","Data":"0825799559f49d5f45e640d862efec0735c7504f44a3954d773d8e6f8225d9e3"} Oct 30 17:51:43.921199 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.921170 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vdlbp" event={"ID":"2ce04018-ba31-464c-8996-967b9100f174","Type":"ContainerStarted","Data":"3866e6ffabee47730af52b48828cedea8224c425c2f6bcbf19d40c024946200e"} Oct 30 17:51:43.928538 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.928505 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xbxvk" event={"ID":"97c8fb7c-e9a6-4eff-9a66-b70848c03a95","Type":"ContainerStarted","Data":"9f8dee39d1523e2e2efed08471bf46398d73d72028d5b0edae754cac4eb95746"} Oct 30 17:51:43.936646 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.936615 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" event={"ID":"ef96f59234cd19c8b7af6a47596c7964","Type":"ContainerStarted","Data":"26edd1c4daca79cc59d6fea149e66df7a414a32235206f891be900636bc3bc1d"} Oct 30 17:51:43.940801 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.940229 2578 generic.go:356] "Generic (PLEG): container finished" podID="49b07006ee51f49de29dbabbdd01d907" containerID="a16a80d5d5df75babf88618464c6c255ad9c9ea0851d381dde490fb683381a6a" exitCode=0 Oct 30 17:51:43.940801 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.940307 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" event={"ID":"49b07006ee51f49de29dbabbdd01d907","Type":"ContainerDied","Data":"a16a80d5d5df75babf88618464c6c255ad9c9ea0851d381dde490fb683381a6a"} Oct 30 17:51:43.947597 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.947517 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sfml" event={"ID":"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881","Type":"ContainerStarted","Data":"c8f683348012bd6873a25ecaf7c515e4200ff75713d9f8105822f8efbec1d83f"} Oct 30 17:51:43.967738 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.967618 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" event={"ID":"d4c590f5-af8b-4994-8b03-7f5cee045767","Type":"ContainerStarted","Data":"0cd0faf02ac1d0545ae1275e67b860caa84ccb64b1c0455eab979dbaad30084b"} Oct 30 17:51:43.972220 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.972123 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wbzt5" event={"ID":"d1e25bb5-1c5b-449e-a7f0-0deb010035ed","Type":"ContainerStarted","Data":"19cd058eb83329fa6bc4277bf0e5e8faa72e400ef58a8faa61470589005876c9"} Oct 30 17:51:43.982762 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:43.982711 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-md4zx" event={"ID":"07abda71-7d42-43e6-9878-90ab7da7d9c4","Type":"ContainerStarted","Data":"3952840a381d0ce2f4744ec7477a9cf0c6c5fae507720774505f073f2d2e8bbb"} Oct 30 17:51:44.025905 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:44.025351 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" podStartSLOduration=2.025329972 podStartE2EDuration="2.025329972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:43.980365345 +0000 UTC m=+3.588059769" watchObservedRunningTime="2026-10-30 17:51:44.025329972 +0000 UTC m=+3.633024399" Oct 30 17:51:44.491176 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:44.491091 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:44.491176 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:44.491144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:44.491408 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:44.491270 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:44.491408 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:44.491328 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret podName:44ce8b1c-6ab2-4f83-a1f6-183cc6cea104 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:46.49131114 +0000 UTC m=+6.099005556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret") pod "global-pull-secret-syncer-fglxs" (UID: "44ce8b1c-6ab2-4f83-a1f6-183cc6cea104") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:44.491408 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:44.491392 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:44.491567 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:44.491424 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs podName:19bd958f-7d20-4da4-818f-06e23e01ab37 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:46.49141337 +0000 UTC m=+6.099107787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs") pod "network-metrics-daemon-dl2lm" (UID: "19bd958f-7d20-4da4-818f-06e23e01ab37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:44.592235 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:44.592199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2j6\" (UniqueName: \"kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6\") pod \"network-check-target-vqltd\" (UID: \"d5d1adae-1728-499a-9d16-ec24f1d69160\") " pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:44.592428 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:44.592411 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:44.592498 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:44.592437 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:44.592498 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:44.592449 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4s2j6 for pod openshift-network-diagnostics/network-check-target-vqltd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:44.592592 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:44.592504 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6 podName:d5d1adae-1728-499a-9d16-ec24f1d69160 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:46.592487105 +0000 UTC m=+6.200181522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4s2j6" (UniqueName: "kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6") pod "network-check-target-vqltd" (UID: "d5d1adae-1728-499a-9d16-ec24f1d69160") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:44.822814 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:44.822693 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:46:42 +0000 UTC" deadline="2028-04-18 12:32:01.42808125 +0000 UTC" Oct 30 17:51:44.822814 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:44.822735 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12858h40m16.605351274s" Oct 30 17:51:44.884317 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:44.882210 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:44.884317 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:44.882346 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:51:44.884317 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:44.882785 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:44.884317 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:44.882890 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:51:45.011725 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:45.011685 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" event={"ID":"49b07006ee51f49de29dbabbdd01d907","Type":"ContainerStarted","Data":"e5f4edf4cfa93f2395a11111ca9f38236dc5a0db0248a3a2fdbdf420a7bf490b"} Oct 30 17:51:45.055440 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:45.055377 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" podStartSLOduration=3.055357492 podStartE2EDuration="3.055357492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:45.055249206 +0000 UTC m=+4.662943631" watchObservedRunningTime="2026-10-30 17:51:45.055357492 +0000 UTC m=+4.663051918" Oct 30 17:51:45.882666 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:45.882412 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:45.882666 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:45.882541 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:51:46.511610 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:46.510801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:46.511610 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:46.510856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:46.511610 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:46.511011 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:46.511610 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:46.511096 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret podName:44ce8b1c-6ab2-4f83-a1f6-183cc6cea104 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:50.511079227 +0000 UTC m=+10.118773633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret") pod "global-pull-secret-syncer-fglxs" (UID: "44ce8b1c-6ab2-4f83-a1f6-183cc6cea104") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:46.511610 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:46.511519 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:46.511610 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:46.511573 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs podName:19bd958f-7d20-4da4-818f-06e23e01ab37 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:50.511557686 +0000 UTC m=+10.119252091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs") pod "network-metrics-daemon-dl2lm" (UID: "19bd958f-7d20-4da4-818f-06e23e01ab37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:46.612265 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:46.611587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2j6\" (UniqueName: \"kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6\") pod \"network-check-target-vqltd\" (UID: \"d5d1adae-1728-499a-9d16-ec24f1d69160\") " pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:46.612265 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:46.611802 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:46.612265 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:46.611825 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:46.612265 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:46.611839 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4s2j6 for pod openshift-network-diagnostics/network-check-target-vqltd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:46.612265 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:46.611895 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6 podName:d5d1adae-1728-499a-9d16-ec24f1d69160 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:50.611877698 +0000 UTC m=+10.219572117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4s2j6" (UniqueName: "kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6") pod "network-check-target-vqltd" (UID: "d5d1adae-1728-499a-9d16-ec24f1d69160") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:46.883315 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:46.882756 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:46.883315 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:46.882891 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:51:46.884110 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:46.883954 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:46.884110 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:46.884064 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:51:47.035437 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:47.034609 2578 generic.go:356] "Generic (PLEG): container finished" podID="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" containerID="522af28188fcd61bec9bf1f3f5ee9a993d21446672c906e96deb48843b1fd4af" exitCode=0 Oct 30 17:51:47.035437 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:47.034659 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sfml" event={"ID":"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881","Type":"ContainerDied","Data":"522af28188fcd61bec9bf1f3f5ee9a993d21446672c906e96deb48843b1fd4af"} Oct 30 17:51:47.882650 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:47.882551 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:47.882806 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:47.882715 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:51:48.041047 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:48.040317 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7gt8q" event={"ID":"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0","Type":"ContainerStarted","Data":"02faf7a157d4dd78bcdd18f01310f2568ba457987888a461ac60fbdbdbfd725c"} Oct 30 17:51:48.048433 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:48.048012 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vdlbp" event={"ID":"2ce04018-ba31-464c-8996-967b9100f174","Type":"ContainerStarted","Data":"eba65ffba703b97e53d8704822b1018016ceb2f377c76a182f8c95c8d3b38e21"} Oct 30 17:51:48.148174 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:48.148083 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vdlbp" podStartSLOduration=3.359968048 podStartE2EDuration="7.148067077s" totalImagesPullingTime="3.788099029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:48.147454543 +0000 UTC m=+7.755148968" watchObservedRunningTime="2026-10-30 17:51:48.148067077 +0000 UTC m=+7.755761501" Oct 30 17:51:48.148354 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:48.148322 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7gt8q" podStartSLOduration=2.729224876 podStartE2EDuration="7.148314536s" totalImagesPullingTime="4.41908966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:48.101307157 +0000 UTC m=+7.709001580" watchObservedRunningTime="2026-10-30 17:51:48.148314536 +0000 UTC m=+7.756008960" Oct 30 17:51:48.885080 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:48.885016 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:48.885245 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:48.885153 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:51:48.885317 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:48.885229 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:48.885365 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:48.885334 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:51:49.053104 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:49.053017 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" event={"ID":"9d6f83f6-600c-4fa7-a20b-6a179c69bffc","Type":"ContainerStarted","Data":"403afc1ca4776c74054638c7c7dfdadc017ee83646f8f67ec5cc5b67c921271c"} Oct 30 17:51:49.882203 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:49.882133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:49.882364 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:49.882299 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:51:50.295892 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:50.295837 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:51:50.548236 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:50.548194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:50.548408 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:50.548286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:50.548408 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:50.548390 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:50.548523 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:50.548448 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs podName:19bd958f-7d20-4da4-818f-06e23e01ab37 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:58.548431284 +0000 UTC m=+18.156125703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs") pod "network-metrics-daemon-dl2lm" (UID: "19bd958f-7d20-4da4-818f-06e23e01ab37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:50.548909 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:50.548861 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:50.549017 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:50.548919 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret podName:44ce8b1c-6ab2-4f83-a1f6-183cc6cea104 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:58.548902386 +0000 UTC m=+18.156596806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret") pod "global-pull-secret-syncer-fglxs" (UID: "44ce8b1c-6ab2-4f83-a1f6-183cc6cea104") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:50.648866 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:50.648769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2j6\" (UniqueName: \"kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6\") pod \"network-check-target-vqltd\" (UID: \"d5d1adae-1728-499a-9d16-ec24f1d69160\") " pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:50.649047 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:50.648979 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:50.649047 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:50.649009 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:50.649047 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:50.649022 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4s2j6 for pod openshift-network-diagnostics/network-check-target-vqltd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:50.649205 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:50.649081 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6 podName:d5d1adae-1728-499a-9d16-ec24f1d69160 nodeName:}" failed. No retries permitted until 2026-10-30 17:51:58.649062495 +0000 UTC m=+18.256756903 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4s2j6" (UniqueName: "kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6") pod "network-check-target-vqltd" (UID: "d5d1adae-1728-499a-9d16-ec24f1d69160") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:50.833055 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:50.832523 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:51:50.295864147Z","UUID":"4dcf4242-1b66-4b81-9c63-495e4aabc1fd","Handler":null,"Name":"","Endpoint":""} Oct 30 17:51:50.842433 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:50.842402 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:51:50.842608 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:50.842455 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:51:50.883499 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:50.883463 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:50.883679 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:50.883510 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:50.883679 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:50.883592 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:51:50.883792 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:50.883688 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:51:51.059987 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:51.059927 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-md4zx" event={"ID":"07abda71-7d42-43e6-9878-90ab7da7d9c4","Type":"ContainerStarted","Data":"ac61507a08edd6c3e5072adaa045a11857c72e8611d3bbae9e28bc6f87c03f5d"} Oct 30 17:51:51.064383 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:51.064298 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" event={"ID":"9d6f83f6-600c-4fa7-a20b-6a179c69bffc","Type":"ContainerStarted","Data":"b28c5ce972bcbfe5a3b840d4ab6e5752d8d48ec988a6160754491e6ca106e341"} Oct 30 17:51:51.066441 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:51.066062 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xbxvk" event={"ID":"97c8fb7c-e9a6-4eff-9a66-b70848c03a95","Type":"ContainerStarted","Data":"e4413879e058d298afba3326a4c95e0e7fb28a262fe895fddf2de8c6983616a6"} Oct 30 17:51:51.150519 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:51.150467 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xbxvk" podStartSLOduration=3.732377139 podStartE2EDuration="11.150446388s" totalImagesPullingTime="7.418069249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:51.150191131 +0000 UTC m=+10.757885555" watchObservedRunningTime="2026-10-30 17:51:51.150446388 +0000 UTC m=+10.758140812" Oct 30 17:51:51.150711 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:51.150564 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-md4zx" podStartSLOduration=2.935018631 podStartE2EDuration="10.150558471s" totalImagesPullingTime="7.21553984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:51.108100441 +0000 UTC m=+10.715794864" watchObservedRunningTime="2026-10-30 17:51:51.150558471 +0000 UTC m=+10.758252894" Oct 30 17:51:51.602850 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:51.602601 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:51:51.603498 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:51.603481 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:51:51.882953 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:51.882866 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:51.883101 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:51.882998 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:51:52.070097 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:52.070060 2578 generic.go:356] "Generic (PLEG): container finished" podID="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" containerID="4206893f44bad5f9224c0a8780dabcd4d9f7a884845cf858c6a9adcd4d03926e" exitCode=0 Oct 30 17:51:52.070259 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:52.070147 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sfml" event={"ID":"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881","Type":"ContainerDied","Data":"4206893f44bad5f9224c0a8780dabcd4d9f7a884845cf858c6a9adcd4d03926e"} Oct 30 17:51:52.073734 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:52.073710 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" event={"ID":"9d6f83f6-600c-4fa7-a20b-6a179c69bffc","Type":"ContainerStarted","Data":"d430db9236b4ad42d5a0cddb0ed5c789ea80d260bd46a1a9984a2de65650ad6f"} Oct 30 17:51:52.882496 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:52.882461 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:52.882496 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:52.882476 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:52.883161 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:52.882603 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:51:52.883161 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:52.882738 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:51:53.078463 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:53.078385 2578 generic.go:356] "Generic (PLEG): container finished" podID="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" containerID="20692f6b55bd7ee409ad0dca5174951273e19b47d9350d55046a0379d1e2714b" exitCode=0 Oct 30 17:51:53.078463 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:53.078441 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sfml" event={"ID":"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881","Type":"ContainerDied","Data":"20692f6b55bd7ee409ad0dca5174951273e19b47d9350d55046a0379d1e2714b"} Oct 30 17:51:53.080351 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:53.080323 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wbzt5" event={"ID":"d1e25bb5-1c5b-449e-a7f0-0deb010035ed","Type":"ContainerStarted","Data":"e8717f78658ec24c682b784123cf97906ddef6b68571b4321693e1ed1fc2eee3"} Oct 30 17:51:53.127694 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:53.127636 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" podStartSLOduration=4.404149126 podStartE2EDuration="12.127617281s" totalImagesPullingTime="7.723468155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:52.180272551 +0000 UTC m=+11.787966978" watchObservedRunningTime="2026-10-30 17:51:53.127617281 +0000 UTC m=+12.735311706" Oct 30 17:51:53.882096 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:53.882059 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:53.882270 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:53.882185 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:51:54.086995 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:54.086962 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dpch6" event={"ID":"5e94fb26-859a-48d2-a493-200543aa1881","Type":"ContainerStarted","Data":"2db4cca49a3218443724352d540fe62a05a2424cf0c363e56ed810cd20b33256"} Oct 30 17:51:54.089500 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:54.089472 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sfml" event={"ID":"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881","Type":"ContainerStarted","Data":"af69146a931f76689cafde2179fe4cc5a32bb150be5580390cd4484689ff8588"} Oct 30 17:51:54.133452 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:54.133346 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wbzt5" podStartSLOduration=6.219594509 podStartE2EDuration="14.13332954s" totalImagesPullingTime="7.913735031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:53.170121401 +0000 UTC m=+12.777815824" watchObservedRunningTime="2026-10-30 17:51:54.13332954 +0000 UTC m=+13.741023966" Oct 30 17:51:54.133650 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:54.133624 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dpch6" podStartSLOduration=3.192767138 podStartE2EDuration="13.133618198s" totalImagesPullingTime="9.94085106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:54.132769486 +0000 UTC m=+13.740463901" watchObservedRunningTime="2026-10-30 17:51:54.133618198 +0000 UTC m=+13.741312625" Oct 30 17:51:54.882216 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:54.882180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:54.882379 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:54.882180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:54.882379 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:54.882311 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:51:54.882630 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:54.882602 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:51:55.094563 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:55.094517 2578 generic.go:356] "Generic (PLEG): container finished" podID="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" containerID="af69146a931f76689cafde2179fe4cc5a32bb150be5580390cd4484689ff8588" exitCode=0 Oct 30 17:51:55.095051 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:55.094592 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sfml" event={"ID":"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881","Type":"ContainerDied","Data":"af69146a931f76689cafde2179fe4cc5a32bb150be5580390cd4484689ff8588"} Oct 30 17:51:55.882308 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:55.882274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:55.882484 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:55.882389 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:51:56.100366 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:56.100331 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sfml" event={"ID":"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881","Type":"ContainerStarted","Data":"55e18ae0b3eb8a4053a504a1f2e7a67f7a444cf735fc8d0437529b2cde94cb5b"} Oct 30 17:51:56.882947 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:56.882901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:56.882947 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:56.882930 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:56.883182 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:56.883030 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:51:56.883182 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:56.883139 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:51:57.105613 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:57.105583 2578 generic.go:356] "Generic (PLEG): container finished" podID="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" containerID="55e18ae0b3eb8a4053a504a1f2e7a67f7a444cf735fc8d0437529b2cde94cb5b" exitCode=0 Oct 30 17:51:57.106025 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:57.105619 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sfml" event={"ID":"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881","Type":"ContainerDied","Data":"55e18ae0b3eb8a4053a504a1f2e7a67f7a444cf735fc8d0437529b2cde94cb5b"} Oct 30 17:51:57.107228 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:57.107205 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" event={"ID":"d4c590f5-af8b-4994-8b03-7f5cee045767","Type":"ContainerStarted","Data":"3bf7b6c3f865e2b9395c110e456571f16ee9a26460447c57c51ca76c3f9ce3e4"} Oct 30 17:51:57.882638 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:57.882428 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:57.882794 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:57.882668 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:51:58.112353 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.112322 2578 generic.go:356] "Generic (PLEG): container finished" podID="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" containerID="9e3e52a68b94c103eaded9942a7bd9f499f985c596d74c9b37a974444566a05e" exitCode=0 Oct 30 17:51:58.112894 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.112408 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sfml" event={"ID":"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881","Type":"ContainerDied","Data":"9e3e52a68b94c103eaded9942a7bd9f499f985c596d74c9b37a974444566a05e"} Oct 30 17:51:58.115330 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.115251 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 17:51:58.115559 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.115541 2578 generic.go:356] "Generic (PLEG): container finished" podID="d4c590f5-af8b-4994-8b03-7f5cee045767" containerID="e6339ce54b85a59f0f0bf4ae406ad5739c82d459b07af86b0ccf0dcc93b9cb10" exitCode=1 Oct 30 17:51:58.115650 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.115578 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" event={"ID":"d4c590f5-af8b-4994-8b03-7f5cee045767","Type":"ContainerStarted","Data":"9cc6cc870af37da6dc0c9b52db91b43be5c15e3e3d427577d7d925bed9c4c3c6"} Oct 30 17:51:58.115650 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.115631 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" event={"ID":"d4c590f5-af8b-4994-8b03-7f5cee045767","Type":"ContainerStarted","Data":"e6b79aa8f2a06a8b31ec00a92d02044e888a87f6215b676798adb9c7a4a3e460"} Oct 30 17:51:58.115754 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.115662 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" event={"ID":"d4c590f5-af8b-4994-8b03-7f5cee045767","Type":"ContainerStarted","Data":"721f1926039921ce66d728ff4fb8ed71ce62e0c43a9c8d92585a9cbbe4cf5218"} Oct 30 17:51:58.115754 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.115696 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" event={"ID":"d4c590f5-af8b-4994-8b03-7f5cee045767","Type":"ContainerStarted","Data":"0756770d96627d1ee4e453df440e8da8c0d22112173dbb39ccc25302412eb4ef"} Oct 30 17:51:58.115754 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.115725 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" event={"ID":"d4c590f5-af8b-4994-8b03-7f5cee045767","Type":"ContainerDied","Data":"e6339ce54b85a59f0f0bf4ae406ad5739c82d459b07af86b0ccf0dcc93b9cb10"} Oct 30 17:51:58.607473 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.607387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:58.607473 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.607429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:58.607631 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:58.607536 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:58.607631 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:58.607538 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:58.607631 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:58.607585 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret podName:44ce8b1c-6ab2-4f83-a1f6-183cc6cea104 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:14.607573279 +0000 UTC m=+34.215267695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret") pod "global-pull-secret-syncer-fglxs" (UID: "44ce8b1c-6ab2-4f83-a1f6-183cc6cea104") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:51:58.607631 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:58.607599 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs podName:19bd958f-7d20-4da4-818f-06e23e01ab37 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:14.607592622 +0000 UTC m=+34.215287023 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs") pod "network-metrics-daemon-dl2lm" (UID: "19bd958f-7d20-4da4-818f-06e23e01ab37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:51:58.708583 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.708548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2j6\" (UniqueName: \"kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6\") pod \"network-check-target-vqltd\" (UID: \"d5d1adae-1728-499a-9d16-ec24f1d69160\") " pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:58.708719 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:58.708661 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:51:58.708719 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:58.708675 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:51:58.708719 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:58.708685 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4s2j6 for pod openshift-network-diagnostics/network-check-target-vqltd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:58.708825 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:58.708728 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6 podName:d5d1adae-1728-499a-9d16-ec24f1d69160 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:14.708716435 +0000 UTC m=+34.316410838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4s2j6" (UniqueName: "kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6") pod "network-check-target-vqltd" (UID: "d5d1adae-1728-499a-9d16-ec24f1d69160") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:51:58.882723 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.882642 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:51:58.882854 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:58.882755 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:51:58.882854 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:58.882808 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:51:58.882924 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:58.882898 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:51:59.120298 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:59.120262 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sfml" event={"ID":"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881","Type":"ContainerStarted","Data":"ec2a1be2c2f725995bd068ef047acd9c4f757f380a12bff8f06b0cbdf94fb3df"} Oct 30 17:51:59.171403 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:59.171312 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2sfml" podStartSLOduration=8.057480747 podStartE2EDuration="18.171299024s" totalImagesPullingTime="10.113818277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:59.171004673 +0000 UTC m=+18.778699097" watchObservedRunningTime="2026-10-30 17:51:59.171299024 +0000 UTC m=+18.778993448" Oct 30 17:51:59.882551 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:51:59.882471 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:51:59.882691 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:51:59.882567 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:52:00.125372 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:00.125343 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 17:52:00.125753 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:00.125719 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" event={"ID":"d4c590f5-af8b-4994-8b03-7f5cee045767","Type":"ContainerStarted","Data":"6b2bcd158d9eb913dc9c10514d3d58d2613ed4332aa4bdf2dca3a354f3e7bf37"} Oct 30 17:52:00.882866 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:00.882839 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:00.883018 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:00.882926 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:52:00.883055 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:00.883033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:00.883185 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:00.883163 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:52:01.883182 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:01.882947 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:01.883664 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:01.883199 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:52:02.131954 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:02.131910 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 17:52:02.132271 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:02.132242 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" event={"ID":"d4c590f5-af8b-4994-8b03-7f5cee045767","Type":"ContainerStarted","Data":"c1426980dfc07a73ff71763860b8931314e6d0076cfc74546c72d5559c55eba6"} Oct 30 17:52:02.132469 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:02.132452 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:02.132548 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:02.132524 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:02.132628 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:02.132611 2578 scope.go:122] "RemoveContainer" containerID="e6339ce54b85a59f0f0bf4ae406ad5739c82d459b07af86b0ccf0dcc93b9cb10" Oct 30 17:52:02.149095 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:02.149073 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:02.149320 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:02.149306 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:02.882579 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:02.882542 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:02.882801 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:02.882560 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:02.882801 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:02.882702 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:52:02.882801 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:02.882781 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:52:03.137322 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:03.137255 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 17:52:03.137725 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:03.137586 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" event={"ID":"d4c590f5-af8b-4994-8b03-7f5cee045767","Type":"ContainerStarted","Data":"c5fe95bcecb4019e849628868da9769a2ae027162aea0a80526082c6dd3271cc"} Oct 30 17:52:03.137725 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:03.137666 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:03.236436 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:03.236381 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" podStartSLOduration=8.401362503 podStartE2EDuration="22.236366998s" totalImagesPullingTime="13.835004495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:03.22148736 +0000 UTC m=+22.829181784" watchObservedRunningTime="2026-10-30 17:52:03.236366998 +0000 UTC m=+22.844061468" Oct 30 17:52:03.682302 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:03.682258 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fglxs"] Oct 30 17:52:03.682489 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:03.682441 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:03.682579 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:03.682552 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:52:03.683008 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:03.682988 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vqltd"] Oct 30 17:52:03.683087 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:03.683071 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:03.683183 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:03.683152 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:52:03.684525 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:03.684503 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dl2lm"] Oct 30 17:52:03.684603 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:03.684590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:03.684676 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:03.684662 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:52:04.139600 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:04.139529 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:04.882478 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:04.882441 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:04.882478 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:04.882441 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:04.882708 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:04.882574 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:52:04.882708 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:04.882686 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:52:05.882468 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:05.882440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:05.882804 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:05.882554 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:52:06.831836 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:06.831793 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:06.832033 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:06.832021 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:06.842390 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:06.842355 2578 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" podUID="d4c590f5-af8b-4994-8b03-7f5cee045767" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 17:52:06.850656 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:06.850629 2578 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" podUID="d4c590f5-af8b-4994-8b03-7f5cee045767" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 17:52:06.883015 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:06.882987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:06.883372 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:06.882987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:06.883372 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:06.883133 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:52:06.883372 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:06.883155 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:52:07.882829 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:07.882791 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:07.883078 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:07.882903 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:52:08.148308 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:08.148281 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:52:08.148499 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:08.148407 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:08.148889 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:08.148871 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:52:08.882467 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:08.882265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:08.882626 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:08.882341 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:08.882626 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:08.882548 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fglxs" podUID="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" Oct 30 17:52:08.882695 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:08.882627 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqltd" podUID="d5d1adae-1728-499a-9d16-ec24f1d69160" Oct 30 17:52:09.882605 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:09.882572 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:09.883015 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:09.882686 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dl2lm" podUID="19bd958f-7d20-4da4-818f-06e23e01ab37" Oct 30 17:52:10.116469 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.116442 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeReady" Oct 30 17:52:10.116605 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.116554 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 17:52:10.200330 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.200251 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2tjnd"] Oct 30 17:52:10.205473 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.205448 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wjjnl"] Oct 30 17:52:10.205623 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.205580 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2tjnd" Oct 30 17:52:10.208905 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.208884 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m"] Oct 30 17:52:10.209093 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.209006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wjjnl" Oct 30 17:52:10.214488 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.214469 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-csr4p"] Oct 30 17:52:10.214609 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.214593 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m" Oct 30 17:52:10.221523 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.221499 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26"] Oct 30 17:52:10.221618 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.221609 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.228669 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.228652 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5zcf9"] Oct 30 17:52:10.228759 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.228745 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:10.235786 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.235768 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt"] Oct 30 17:52:10.235893 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.235880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.241854 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.241839 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.242820 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.242801 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.242959 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.242931 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jjhvv\"" Oct 30 17:52:10.243013 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.242958 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm"] Oct 30 17:52:10.243051 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.243021 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:10.246257 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.246237 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-f6d947f75-ln5z8"] Oct 30 17:52:10.246949 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.246909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:10.250029 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.250012 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6fncr"] Oct 30 17:52:10.250123 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.250114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.255465 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.255443 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:10.257484 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.257461 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2tjnd"] Oct 30 17:52:10.268005 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.267980 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t5n25"] Oct 30 17:52:10.273751 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.273732 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m"] Oct 30 17:52:10.273848 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.273769 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26"] Oct 30 17:52:10.273848 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.273792 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt"] Oct 30 17:52:10.273848 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.273821 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5zcf9"] Oct 30 17:52:10.273848 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.273842 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t5n25" Oct 30 17:52:10.276503 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.276468 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wjjnl"] Oct 30 17:52:10.293355 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.293333 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtlqq\" (UniqueName: \"kubernetes.io/projected/06498a3b-c75b-4bf5-9fc7-beced86dbfc5-kube-api-access-gtlqq\") pod \"kas-connection-checker-685895c7c9-wjjnl\" (UID: \"06498a3b-c75b-4bf5-9fc7-beced86dbfc5\") " pod="kube-system/kas-connection-checker-685895c7c9-wjjnl" Oct 30 17:52:10.293455 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.293365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xx66\" (UniqueName: \"kubernetes.io/projected/b6956a31-29a1-4cd7-bc8b-9cda43dc2e73-kube-api-access-4xx66\") pod \"kas-connection-checker-685895c7c9-2tjnd\" (UID: \"b6956a31-29a1-4cd7-bc8b-9cda43dc2e73\") " pod="kube-system/kas-connection-checker-685895c7c9-2tjnd" Oct 30 17:52:10.293455 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.293390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dxbs\" (UniqueName: \"kubernetes.io/projected/ba65a282-0992-49ce-b4bc-108c8f69016d-kube-api-access-2dxbs\") pod \"volume-data-source-validator-58c9f5f46d-hpv5m\" (UID: \"ba65a282-0992-49ce-b4bc-108c8f69016d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m" Oct 30 17:52:10.311383 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.310951 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm"] Oct 30 17:52:10.313575 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.313550 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t5n25"] Oct 30 17:52:10.314477 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.314449 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f6d947f75-ln5z8"] Oct 30 17:52:10.315176 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.315145 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6fncr"] Oct 30 17:52:10.315765 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.315746 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-csr4p"] Oct 30 17:52:10.346556 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.346519 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 17:52:10.346556 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.346526 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 17:52:10.346724 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.346574 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.346838 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.346825 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.347015 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347000 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 17:52:10.347097 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347023 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.347097 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347061 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.347097 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347082 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 17:52:10.347210 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347115 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 17:52:10.347210 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347122 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 17:52:10.347210 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347123 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 17:52:10.347210 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347138 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.347210 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347185 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.347396 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347224 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.347396 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347284 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.347396 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347287 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cwftc\"" Oct 30 17:52:10.347608 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347593 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.347831 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347659 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.347880 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347838 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:52:10.347880 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347846 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-99rd9\"" Oct 30 17:52:10.347880 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.347848 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 17:52:10.352515 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.352498 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 17:52:10.360317 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.360297 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 17:52:10.360402 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.360299 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 17:52:10.360402 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.360327 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:52:10.360402 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.360343 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 17:52:10.360402 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.360375 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:52:10.360668 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.360657 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6xmc6\"" Oct 30 17:52:10.360765 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.360751 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vnlvf\"" Oct 30 17:52:10.360807 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.360764 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.360807 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.360752 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.360876 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.360810 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.361208 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.361194 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 17:52:10.361517 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.361503 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.367024 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.366962 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 17:52:10.367024 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.366993 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.375763 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.375739 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 17:52:10.375763 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.375760 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.375876 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.375813 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 17:52:10.394064 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4aa5e79-72a0-4b20-a639-ac9b61be4759-serving-cert\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:10.394190 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdl6\" (UniqueName: \"kubernetes.io/projected/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-kube-api-access-bwdl6\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:10.394190 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xx66\" (UniqueName: \"kubernetes.io/projected/b6956a31-29a1-4cd7-bc8b-9cda43dc2e73-kube-api-access-4xx66\") pod \"kas-connection-checker-685895c7c9-2tjnd\" (UID: \"b6956a31-29a1-4cd7-bc8b-9cda43dc2e73\") " pod="kube-system/kas-connection-checker-685895c7c9-2tjnd" Oct 30 17:52:10.394282 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:10.394282 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmtbq\" (UniqueName: \"kubernetes.io/projected/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-kube-api-access-pmtbq\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:10.394282 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6bn\" (UniqueName: \"kubernetes.io/projected/4260cb2f-efee-4b45-9f73-a48757f3ce75-kube-api-access-zr6bn\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.394419 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15f0fc8a-91c4-404a-91ec-b39914e83cae-serving-cert\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.394419 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fb42c05-fb07-4dae-975b-71d45dd52e03-trusted-ca\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.394492 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394443 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-config-volume\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:10.394492 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-tmp-dir\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:10.394562 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:10.394562 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67r2z\" (UniqueName: \"kubernetes.io/projected/4fb42c05-fb07-4dae-975b-71d45dd52e03-kube-api-access-67r2z\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.394639 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394576 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88w5h\" (UniqueName: \"kubernetes.io/projected/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-kube-api-access-88w5h\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:10.394639 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4fb42c05-fb07-4dae-975b-71d45dd52e03-config\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.394734 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.394734 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1aa6b58-d1c3-40a0-873c-86b44c438fac-cert\") pod \"ingress-canary-t5n25\" (UID: \"c1aa6b58-d1c3-40a0-873c-86b44c438fac\") " pod="openshift-ingress-canary/ingress-canary-t5n25" Oct 30 17:52:10.394829 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:10.394829 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:10.394905 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394830 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.394905 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtlqq\" (UniqueName: \"kubernetes.io/projected/06498a3b-c75b-4bf5-9fc7-beced86dbfc5-kube-api-access-gtlqq\") pod \"kas-connection-checker-685895c7c9-wjjnl\" (UID: \"06498a3b-c75b-4bf5-9fc7-beced86dbfc5\") " pod="kube-system/kas-connection-checker-685895c7c9-wjjnl" Oct 30 17:52:10.394988 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394912 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-stats-auth\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.394988 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394929 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/15f0fc8a-91c4-404a-91ec-b39914e83cae-snapshots\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.395077 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.394983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dxbs\" (UniqueName: \"kubernetes.io/projected/ba65a282-0992-49ce-b4bc-108c8f69016d-kube-api-access-2dxbs\") pod \"volume-data-source-validator-58c9f5f46d-hpv5m\" (UID: \"ba65a282-0992-49ce-b4bc-108c8f69016d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m" Oct 30 17:52:10.395077 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.395011 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:10.395077 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.395035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pq9\" (UniqueName: \"kubernetes.io/projected/c1aa6b58-d1c3-40a0-873c-86b44c438fac-kube-api-access-j9pq9\") pod \"ingress-canary-t5n25\" (UID: \"c1aa6b58-d1c3-40a0-873c-86b44c438fac\") " pod="openshift-ingress-canary/ingress-canary-t5n25" Oct 30 17:52:10.395077 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.395056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15f0fc8a-91c4-404a-91ec-b39914e83cae-tmp\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.395238 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.395078 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9756\" (UniqueName: \"kubernetes.io/projected/15f0fc8a-91c4-404a-91ec-b39914e83cae-kube-api-access-w9756\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.395238 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.395140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-default-certificate\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.395238 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.395177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4fb42c05-fb07-4dae-975b-71d45dd52e03-serving-cert\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.395323 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.395257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f0fc8a-91c4-404a-91ec-b39914e83cae-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.395323 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.395307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f0fc8a-91c4-404a-91ec-b39914e83cae-service-ca-bundle\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.395386 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.395328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4aa5e79-72a0-4b20-a639-ac9b61be4759-config\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:10.395386 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.395347 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf98n\" (UniqueName: \"kubernetes.io/projected/e4aa5e79-72a0-4b20-a639-ac9b61be4759-kube-api-access-nf98n\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:10.437273 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.437245 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtlqq\" (UniqueName: \"kubernetes.io/projected/06498a3b-c75b-4bf5-9fc7-beced86dbfc5-kube-api-access-gtlqq\") pod \"kas-connection-checker-685895c7c9-wjjnl\" (UID: \"06498a3b-c75b-4bf5-9fc7-beced86dbfc5\") " pod="kube-system/kas-connection-checker-685895c7c9-wjjnl" Oct 30 17:52:10.439898 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.439870 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dxbs\" (UniqueName: \"kubernetes.io/projected/ba65a282-0992-49ce-b4bc-108c8f69016d-kube-api-access-2dxbs\") pod \"volume-data-source-validator-58c9f5f46d-hpv5m\" (UID: \"ba65a282-0992-49ce-b4bc-108c8f69016d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m" Oct 30 17:52:10.441205 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.441188 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xx66\" (UniqueName: \"kubernetes.io/projected/b6956a31-29a1-4cd7-bc8b-9cda43dc2e73-kube-api-access-4xx66\") pod \"kas-connection-checker-685895c7c9-2tjnd\" (UID: \"b6956a31-29a1-4cd7-bc8b-9cda43dc2e73\") " pod="kube-system/kas-connection-checker-685895c7c9-2tjnd" Oct 30 17:52:10.495600 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.495571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:10.495600 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.495600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pq9\" (UniqueName: \"kubernetes.io/projected/c1aa6b58-d1c3-40a0-873c-86b44c438fac-kube-api-access-j9pq9\") pod \"ingress-canary-t5n25\" (UID: \"c1aa6b58-d1c3-40a0-873c-86b44c438fac\") " pod="openshift-ingress-canary/ingress-canary-t5n25" Oct 30 17:52:10.495763 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.495616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15f0fc8a-91c4-404a-91ec-b39914e83cae-tmp\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.495763 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:10.495684 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:52:10.495763 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.495728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9756\" (UniqueName: \"kubernetes.io/projected/15f0fc8a-91c4-404a-91ec-b39914e83cae-kube-api-access-w9756\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.495763 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:10.495735 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls podName:867cb674-bd6b-4ca7-98a6-fa31bdaa132d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:10.995720088 +0000 UTC m=+30.603414490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z2t26" (UID: "867cb674-bd6b-4ca7-98a6-fa31bdaa132d") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:52:10.495986 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.495770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-default-certificate\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.495986 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.495801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4fb42c05-fb07-4dae-975b-71d45dd52e03-serving-cert\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.495986 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.495956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f0fc8a-91c4-404a-91ec-b39914e83cae-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.496115 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.495989 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15f0fc8a-91c4-404a-91ec-b39914e83cae-tmp\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.496115 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f0fc8a-91c4-404a-91ec-b39914e83cae-service-ca-bundle\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.496115 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4aa5e79-72a0-4b20-a639-ac9b61be4759-config\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:10.496253 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496115 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf98n\" (UniqueName: \"kubernetes.io/projected/e4aa5e79-72a0-4b20-a639-ac9b61be4759-kube-api-access-nf98n\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:10.496253 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4aa5e79-72a0-4b20-a639-ac9b61be4759-serving-cert\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:10.496253 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdl6\" (UniqueName: \"kubernetes.io/projected/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-kube-api-access-bwdl6\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:10.496447 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:10.496447 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtbq\" (UniqueName: \"kubernetes.io/projected/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-kube-api-access-pmtbq\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:10.496447 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6bn\" (UniqueName: \"kubernetes.io/projected/4260cb2f-efee-4b45-9f73-a48757f3ce75-kube-api-access-zr6bn\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.496598 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15f0fc8a-91c4-404a-91ec-b39914e83cae-serving-cert\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.496598 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fb42c05-fb07-4dae-975b-71d45dd52e03-trusted-ca\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.496598 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-config-volume\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:10.496598 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-tmp-dir\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:10.496598 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496543 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f0fc8a-91c4-404a-91ec-b39914e83cae-service-ca-bundle\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.496598 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:10.496888 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67r2z\" (UniqueName: \"kubernetes.io/projected/4fb42c05-fb07-4dae-975b-71d45dd52e03-kube-api-access-67r2z\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.496888 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88w5h\" (UniqueName: \"kubernetes.io/projected/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-kube-api-access-88w5h\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:10.496888 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4fb42c05-fb07-4dae-975b-71d45dd52e03-config\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.496888 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496722 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4aa5e79-72a0-4b20-a639-ac9b61be4759-config\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:10.496888 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:10.496831 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:10.496888 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:10.496883 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls podName:7cf0aaaf-7ca8-44b7-afad-829e4455e56a nodeName:}" failed. No retries permitted until 2026-10-30 17:52:10.996866093 +0000 UTC m=+30.604560512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls") pod "dns-default-6fncr" (UID: "7cf0aaaf-7ca8-44b7-afad-829e4455e56a") : secret "dns-default-metrics-tls" not found Oct 30 17:52:10.497207 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.496968 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-tmp-dir\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:10.497207 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.497013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.497207 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.497038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1aa6b58-d1c3-40a0-873c-86b44c438fac-cert\") pod \"ingress-canary-t5n25\" (UID: \"c1aa6b58-d1c3-40a0-873c-86b44c438fac\") " pod="openshift-ingress-canary/ingress-canary-t5n25" Oct 30 17:52:10.497207 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.497070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:10.497207 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.497097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:10.497207 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.497135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.497207 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.497169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-stats-auth\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.497207 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.497195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/15f0fc8a-91c4-404a-91ec-b39914e83cae-snapshots\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.497207 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.497199 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f0fc8a-91c4-404a-91ec-b39914e83cae-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.497627 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:10.497308 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:52:10.497627 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:10.497356 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:10.997340613 +0000 UTC m=+30.605035029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : secret "router-metrics-certs-default" not found Oct 30 17:52:10.497627 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.497402 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4fb42c05-fb07-4dae-975b-71d45dd52e03-config\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.497627 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:10.497430 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:10.997419205 +0000 UTC m=+30.605113622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : configmap references non-existent config key: service-ca.crt Oct 30 17:52:10.497627 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:10.497481 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:52:10.497627 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:10.497539 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls podName:1f8536c2-cd14-4d88-ab1d-63a2f032aeb5 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:10.997524304 +0000 UTC m=+30.605218720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r7rxt" (UID: "1f8536c2-cd14-4d88-ab1d-63a2f032aeb5") : secret "samples-operator-tls" not found Oct 30 17:52:10.497924 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.497770 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:10.498094 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.498072 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fb42c05-fb07-4dae-975b-71d45dd52e03-trusted-ca\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.498245 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.498221 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:10.498411 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.498386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-config-volume\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:10.498599 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.498570 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/15f0fc8a-91c4-404a-91ec-b39914e83cae-snapshots\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.498837 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.498811 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-default-certificate\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.499400 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.499381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15f0fc8a-91c4-404a-91ec-b39914e83cae-serving-cert\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.499609 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.499591 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4aa5e79-72a0-4b20-a639-ac9b61be4759-serving-cert\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:10.500104 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.500088 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-stats-auth\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.500226 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.500212 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4fb42c05-fb07-4dae-975b-71d45dd52e03-serving-cert\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.500679 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.500666 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1aa6b58-d1c3-40a0-873c-86b44c438fac-cert\") pod \"ingress-canary-t5n25\" (UID: \"c1aa6b58-d1c3-40a0-873c-86b44c438fac\") " pod="openshift-ingress-canary/ingress-canary-t5n25" Oct 30 17:52:10.522211 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.522190 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2tjnd" Oct 30 17:52:10.532824 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.532796 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wjjnl" Oct 30 17:52:10.535591 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.535567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdl6\" (UniqueName: \"kubernetes.io/projected/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-kube-api-access-bwdl6\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:10.535778 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.535751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf98n\" (UniqueName: \"kubernetes.io/projected/e4aa5e79-72a0-4b20-a639-ac9b61be4759-kube-api-access-nf98n\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:10.535916 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.535899 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6bn\" (UniqueName: \"kubernetes.io/projected/4260cb2f-efee-4b45-9f73-a48757f3ce75-kube-api-access-zr6bn\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:10.536270 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.536244 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67r2z\" (UniqueName: \"kubernetes.io/projected/4fb42c05-fb07-4dae-975b-71d45dd52e03-kube-api-access-67r2z\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.536270 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.536257 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtbq\" (UniqueName: \"kubernetes.io/projected/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-kube-api-access-pmtbq\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:10.536473 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.536455 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9756\" (UniqueName: \"kubernetes.io/projected/15f0fc8a-91c4-404a-91ec-b39914e83cae-kube-api-access-w9756\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.538921 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.538898 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88w5h\" (UniqueName: \"kubernetes.io/projected/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-kube-api-access-88w5h\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:10.539362 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.539347 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m" Oct 30 17:52:10.539655 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.539636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pq9\" (UniqueName: \"kubernetes.io/projected/c1aa6b58-d1c3-40a0-873c-86b44c438fac-kube-api-access-j9pq9\") pod \"ingress-canary-t5n25\" (UID: \"c1aa6b58-d1c3-40a0-873c-86b44c438fac\") " pod="openshift-ingress-canary/ingress-canary-t5n25" Oct 30 17:52:10.544762 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.544746 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:10.551410 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.551388 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:10.567250 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.567220 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:10.586442 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.586166 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t5n25" Oct 30 17:52:10.883778 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.883744 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:10.884323 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.883847 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:10.903662 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.903605 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:52:10.903662 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.903612 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ddhr5\"" Oct 30 17:52:10.906108 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.906094 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:52:10.907733 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.907641 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m"] Oct 30 17:52:10.911469 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.911449 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-csr4p"] Oct 30 17:52:10.913534 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:52:10.913516 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba65a282_0992_49ce_b4bc_108c8f69016d.slice/crio-2cdaadf0279074953b6cff64d8032e4817e8b7d1b9e297ce3e09dbab274dd9a0 WatchSource:0}: Error finding container 2cdaadf0279074953b6cff64d8032e4817e8b7d1b9e297ce3e09dbab274dd9a0: Status 404 returned error can't find the container with id 2cdaadf0279074953b6cff64d8032e4817e8b7d1b9e297ce3e09dbab274dd9a0 Oct 30 17:52:10.917007 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:52:10.916980 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f0fc8a_91c4_404a_91ec_b39914e83cae.slice/crio-ff8e2d857292e8512cc53593b73ef901f6c336dc2f9a9b24e074830970341052 WatchSource:0}: Error finding container ff8e2d857292e8512cc53593b73ef901f6c336dc2f9a9b24e074830970341052: Status 404 returned error can't find the container with id ff8e2d857292e8512cc53593b73ef901f6c336dc2f9a9b24e074830970341052 Oct 30 17:52:10.921185 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.921156 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5zcf9"] Oct 30 17:52:10.924985 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:52:10.924964 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb42c05_fb07_4dae_975b_71d45dd52e03.slice/crio-1a672c72572bbbc22e25fbf1853e3e557e72d76e81375d5db4e9e71412af9561 WatchSource:0}: Error finding container 1a672c72572bbbc22e25fbf1853e3e557e72d76e81375d5db4e9e71412af9561: Status 404 returned error can't find the container with id 1a672c72572bbbc22e25fbf1853e3e557e72d76e81375d5db4e9e71412af9561 Oct 30 17:52:10.936546 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.936524 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2tjnd"] Oct 30 17:52:10.940021 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:52:10.939994 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6956a31_29a1_4cd7_bc8b_9cda43dc2e73.slice/crio-3a9bbea95be20bac1e1bb2c2164a2994870d7dee69975397b9cdf1131560a066 WatchSource:0}: Error finding container 3a9bbea95be20bac1e1bb2c2164a2994870d7dee69975397b9cdf1131560a066: Status 404 returned error can't find the container with id 3a9bbea95be20bac1e1bb2c2164a2994870d7dee69975397b9cdf1131560a066 Oct 30 17:52:10.943272 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.943201 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wjjnl"] Oct 30 17:52:10.945450 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.945367 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm"] Oct 30 17:52:10.945450 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:10.945412 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t5n25"] Oct 30 17:52:10.949850 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:52:10.949824 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06498a3b_c75b_4bf5_9fc7_beced86dbfc5.slice/crio-0d11ff40ffeed3a7b885575145b1a5889d89641ddf5c5dc4bde2ebf43e652912 WatchSource:0}: Error finding container 0d11ff40ffeed3a7b885575145b1a5889d89641ddf5c5dc4bde2ebf43e652912: Status 404 returned error can't find the container with id 0d11ff40ffeed3a7b885575145b1a5889d89641ddf5c5dc4bde2ebf43e652912 Oct 30 17:52:10.950697 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:52:10.950661 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1aa6b58_d1c3_40a0_873c_86b44c438fac.slice/crio-ed98329f6d9d0ac3f0bb6062d734f0d64e2e785e26f672905bb7532f1de9b0f8 WatchSource:0}: Error finding container ed98329f6d9d0ac3f0bb6062d734f0d64e2e785e26f672905bb7532f1de9b0f8: Status 404 returned error can't find the container with id ed98329f6d9d0ac3f0bb6062d734f0d64e2e785e26f672905bb7532f1de9b0f8 Oct 30 17:52:10.951190 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:52:10.951169 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4aa5e79_72a0_4b20_a639_ac9b61be4759.slice/crio-2f5d511d548c2fdefd45d8bdd0b8065dfcfacab0f31ac1c49fd7a36f41b150e3 WatchSource:0}: Error finding container 2f5d511d548c2fdefd45d8bdd0b8065dfcfacab0f31ac1c49fd7a36f41b150e3: Status 404 returned error can't find the container with id 2f5d511d548c2fdefd45d8bdd0b8065dfcfacab0f31ac1c49fd7a36f41b150e3 Oct 30 17:52:11.001715 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.001669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:11.001873 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.001730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:11.001873 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.001763 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:11.001873 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.001793 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:11.002074 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:11.001968 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:52:11.002131 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:11.002082 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:11.002185 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:11.002163 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:52:11.002266 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:11.002247 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:12.002226352 +0000 UTC m=+31.609920754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : secret "router-metrics-certs-default" not found Oct 30 17:52:11.002366 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:11.002352 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls podName:7cf0aaaf-7ca8-44b7-afad-829e4455e56a nodeName:}" failed. No retries permitted until 2026-10-30 17:52:12.002340737 +0000 UTC m=+31.610035151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls") pod "dns-default-6fncr" (UID: "7cf0aaaf-7ca8-44b7-afad-829e4455e56a") : secret "dns-default-metrics-tls" not found Oct 30 17:52:11.002433 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:11.002389 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls podName:1f8536c2-cd14-4d88-ab1d-63a2f032aeb5 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:12.002379664 +0000 UTC m=+31.610074067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r7rxt" (UID: "1f8536c2-cd14-4d88-ab1d-63a2f032aeb5") : secret "samples-operator-tls" not found Oct 30 17:52:11.002433 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:11.002406 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:12.002396446 +0000 UTC m=+31.610090855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : configmap references non-existent config key: service-ca.crt Oct 30 17:52:11.002433 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.002426 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:11.002591 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:11.002522 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:52:11.002591 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:11.002557 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls podName:867cb674-bd6b-4ca7-98a6-fa31bdaa132d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:12.002547148 +0000 UTC m=+31.610241565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z2t26" (UID: "867cb674-bd6b-4ca7-98a6-fa31bdaa132d") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:52:11.003960 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.003915 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:52:11.004054 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.004041 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-x8qzn\"" Oct 30 17:52:11.156031 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.155970 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2tjnd" event={"ID":"b6956a31-29a1-4cd7-bc8b-9cda43dc2e73","Type":"ContainerStarted","Data":"4509967cc6b5001b071b2755bd0d7b4e1b55c57b59396e44d3621b065234b8ec"} Oct 30 17:52:11.156031 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.156020 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2tjnd" event={"ID":"b6956a31-29a1-4cd7-bc8b-9cda43dc2e73","Type":"ContainerStarted","Data":"3a9bbea95be20bac1e1bb2c2164a2994870d7dee69975397b9cdf1131560a066"} Oct 30 17:52:11.157306 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.157276 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t5n25" event={"ID":"c1aa6b58-d1c3-40a0-873c-86b44c438fac","Type":"ContainerStarted","Data":"ed98329f6d9d0ac3f0bb6062d734f0d64e2e785e26f672905bb7532f1de9b0f8"} Oct 30 17:52:11.158532 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.158504 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" event={"ID":"4fb42c05-fb07-4dae-975b-71d45dd52e03","Type":"ContainerStarted","Data":"1a672c72572bbbc22e25fbf1853e3e557e72d76e81375d5db4e9e71412af9561"} Oct 30 17:52:11.159651 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.159631 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m" event={"ID":"ba65a282-0992-49ce-b4bc-108c8f69016d","Type":"ContainerStarted","Data":"2cdaadf0279074953b6cff64d8032e4817e8b7d1b9e297ce3e09dbab274dd9a0"} Oct 30 17:52:11.161168 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.161145 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wjjnl" event={"ID":"06498a3b-c75b-4bf5-9fc7-beced86dbfc5","Type":"ContainerStarted","Data":"a21131bc5e7f0b9ffcc04f7eb0a55012de61ff73fa42d0fc85aecffa8a433d17"} Oct 30 17:52:11.161257 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.161181 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wjjnl" event={"ID":"06498a3b-c75b-4bf5-9fc7-beced86dbfc5","Type":"ContainerStarted","Data":"0d11ff40ffeed3a7b885575145b1a5889d89641ddf5c5dc4bde2ebf43e652912"} Oct 30 17:52:11.162316 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.162289 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" event={"ID":"e4aa5e79-72a0-4b20-a639-ac9b61be4759","Type":"ContainerStarted","Data":"2f5d511d548c2fdefd45d8bdd0b8065dfcfacab0f31ac1c49fd7a36f41b150e3"} Oct 30 17:52:11.163340 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.163314 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-csr4p" event={"ID":"15f0fc8a-91c4-404a-91ec-b39914e83cae","Type":"ContainerStarted","Data":"ff8e2d857292e8512cc53593b73ef901f6c336dc2f9a9b24e074830970341052"} Oct 30 17:52:11.199334 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.199282 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-2tjnd" podStartSLOduration=399.199265392 podStartE2EDuration="6m39.199265392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:11.198558789 +0000 UTC m=+30.806253212" watchObservedRunningTime="2026-10-30 17:52:11.199265392 +0000 UTC m=+30.806959817" Oct 30 17:52:11.242462 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.242411 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wjjnl" podStartSLOduration=399.242398735 podStartE2EDuration="6m39.242398735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:11.240702289 +0000 UTC m=+30.848396713" watchObservedRunningTime="2026-10-30 17:52:11.242398735 +0000 UTC m=+30.850093159" Oct 30 17:52:11.883521 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.882387 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:11.900967 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.900431 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zlfnf\"" Oct 30 17:52:11.901334 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:11.901004 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:52:12.012446 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:12.012420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:12.012601 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:12.012456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:12.012601 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:12.012479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:12.012601 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:12.012503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:12.012601 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:12.012537 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:12.012764 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:12.012675 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:52:12.012764 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:12.012719 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls podName:867cb674-bd6b-4ca7-98a6-fa31bdaa132d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:14.012706781 +0000 UTC m=+33.620401196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z2t26" (UID: "867cb674-bd6b-4ca7-98a6-fa31bdaa132d") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:52:12.013077 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:12.013060 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:12.013126 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:12.013113 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls podName:7cf0aaaf-7ca8-44b7-afad-829e4455e56a nodeName:}" failed. No retries permitted until 2026-10-30 17:52:14.013096788 +0000 UTC m=+33.620791205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls") pod "dns-default-6fncr" (UID: "7cf0aaaf-7ca8-44b7-afad-829e4455e56a") : secret "dns-default-metrics-tls" not found Oct 30 17:52:12.013212 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:12.013202 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:14.013194136 +0000 UTC m=+33.620888552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : configmap references non-existent config key: service-ca.crt Oct 30 17:52:12.013320 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:12.013298 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:52:12.013363 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:12.013348 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:52:12.013395 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:12.013364 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls podName:1f8536c2-cd14-4d88-ab1d-63a2f032aeb5 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:14.013346356 +0000 UTC m=+33.621040781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r7rxt" (UID: "1f8536c2-cd14-4d88-ab1d-63a2f032aeb5") : secret "samples-operator-tls" not found Oct 30 17:52:12.013395 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:12.013386 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:14.013374186 +0000 UTC m=+33.621068621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : secret "router-metrics-certs-default" not found Oct 30 17:52:13.173841 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:13.173798 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m" event={"ID":"ba65a282-0992-49ce-b4bc-108c8f69016d","Type":"ContainerStarted","Data":"9b0d0724c98ec0af0893638dedc963aa57ca5f23d3add9ec81069cfc0907a748"} Oct 30 17:52:13.218483 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:13.218424 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m" podStartSLOduration=366.323983287 podStartE2EDuration="6m8.21840574s" totalImagesPullingTime="1.894422453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:13.217001142 +0000 UTC m=+32.824695579" watchObservedRunningTime="2026-10-30 17:52:13.21840574 +0000 UTC m=+32.826100210" Oct 30 17:52:14.031805 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.031727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:14.031805 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.031781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:14.032037 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.031814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:14.032037 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.031841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:14.032037 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.031879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:14.032037 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:14.031909 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:52:14.032037 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:14.031910 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:14.032037 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:14.031994 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:52:14.032037 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:14.032003 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:52:14.032339 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:14.031992 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.031975087 +0000 UTC m=+37.639669511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : secret "router-metrics-certs-default" not found Oct 30 17:52:14.032339 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:14.032075 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls podName:7cf0aaaf-7ca8-44b7-afad-829e4455e56a nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.032060921 +0000 UTC m=+37.639755324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls") pod "dns-default-6fncr" (UID: "7cf0aaaf-7ca8-44b7-afad-829e4455e56a") : secret "dns-default-metrics-tls" not found Oct 30 17:52:14.032339 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:14.032095 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.03208431 +0000 UTC m=+37.639778714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : configmap references non-existent config key: service-ca.crt Oct 30 17:52:14.032339 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:14.032114 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls podName:1f8536c2-cd14-4d88-ab1d-63a2f032aeb5 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.032106065 +0000 UTC m=+37.639800480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r7rxt" (UID: "1f8536c2-cd14-4d88-ab1d-63a2f032aeb5") : secret "samples-operator-tls" not found Oct 30 17:52:14.032339 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:14.032131 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls podName:867cb674-bd6b-4ca7-98a6-fa31bdaa132d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.032123 +0000 UTC m=+37.639817408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z2t26" (UID: "867cb674-bd6b-4ca7-98a6-fa31bdaa132d") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:52:14.635917 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.635876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:14.636394 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.636033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:14.636394 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:14.636036 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:52:14.636394 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:14.636196 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs podName:19bd958f-7d20-4da4-818f-06e23e01ab37 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:46.636176866 +0000 UTC m=+66.243871283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs") pod "network-metrics-daemon-dl2lm" (UID: "19bd958f-7d20-4da4-818f-06e23e01ab37") : secret "metrics-daemon-secret" not found Oct 30 17:52:14.638674 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.638646 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:14.736906 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.736870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2j6\" (UniqueName: \"kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6\") pod \"network-check-target-vqltd\" (UID: \"d5d1adae-1728-499a-9d16-ec24f1d69160\") " pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:14.739582 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.739552 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s2j6\" (UniqueName: \"kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6\") pod \"network-check-target-vqltd\" (UID: \"d5d1adae-1728-499a-9d16-ec24f1d69160\") " pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:14.783119 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.775612 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh"] Oct 30 17:52:14.797765 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.797738 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh"] Oct 30 17:52:14.797855 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.797846 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh" Oct 30 17:52:14.817957 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.817920 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:14.823381 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.823361 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:14.892496 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.892468 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 17:52:14.892599 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.892523 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:14.892671 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.892596 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cgqqv\"" Oct 30 17:52:14.938956 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:14.938727 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv255\" (UniqueName: \"kubernetes.io/projected/3d610c80-f954-4f69-ab9a-e75c56c98652-kube-api-access-gv255\") pod \"migrator-66df5d496b-n52mh\" (UID: \"3d610c80-f954-4f69-ab9a-e75c56c98652\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh" Oct 30 17:52:15.039638 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.039599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv255\" (UniqueName: \"kubernetes.io/projected/3d610c80-f954-4f69-ab9a-e75c56c98652-kube-api-access-gv255\") pod \"migrator-66df5d496b-n52mh\" (UID: \"3d610c80-f954-4f69-ab9a-e75c56c98652\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh" Oct 30 17:52:15.065416 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.065383 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv255\" (UniqueName: \"kubernetes.io/projected/3d610c80-f954-4f69-ab9a-e75c56c98652-kube-api-access-gv255\") pod \"migrator-66df5d496b-n52mh\" (UID: \"3d610c80-f954-4f69-ab9a-e75c56c98652\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh" Oct 30 17:52:15.123949 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.123898 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sw2jb"] Oct 30 17:52:15.134687 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.134657 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh" Oct 30 17:52:15.145317 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.145259 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.147632 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.147608 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vqltd"] Oct 30 17:52:15.151775 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:52:15.151749 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d1adae_1728_499a_9d16_ec24f1d69160.slice/crio-06ec5fb6cb500a01567f6fcd99b57082fda5e62cb8a7adadbb3af9853bd1f97f WatchSource:0}: Error finding container 06ec5fb6cb500a01567f6fcd99b57082fda5e62cb8a7adadbb3af9853bd1f97f: Status 404 returned error can't find the container with id 06ec5fb6cb500a01567f6fcd99b57082fda5e62cb8a7adadbb3af9853bd1f97f Oct 30 17:52:15.169126 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.169102 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sw2jb"] Oct 30 17:52:15.171266 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.171244 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cxbhw\"" Oct 30 17:52:15.172550 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.172522 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:52:15.183912 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.183884 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t5n25" event={"ID":"c1aa6b58-d1c3-40a0-873c-86b44c438fac","Type":"ContainerStarted","Data":"995b0da06605d9e458a599ae7f6272f41faa66097dcbbb026719d17fb9c0ac99"} Oct 30 17:52:15.187297 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.187275 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fglxs"] Oct 30 17:52:15.188652 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.188631 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-x2wf2-g8lv5"] Oct 30 17:52:15.191713 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:52:15.191689 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ce8b1c_6ab2_4f83_a1f6_183cc6cea104.slice/crio-0c7d6cbe457d11143c85e47449fd7800aceaf613a7a8012fe59ca7b1299a4848 WatchSource:0}: Error finding container 0c7d6cbe457d11143c85e47449fd7800aceaf613a7a8012fe59ca7b1299a4848: Status 404 returned error can't find the container with id 0c7d6cbe457d11143c85e47449fd7800aceaf613a7a8012fe59ca7b1299a4848 Oct 30 17:52:15.209536 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.207992 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.209536 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.208829 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/0.log" Oct 30 17:52:15.209536 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.208906 2578 generic.go:356] "Generic (PLEG): container finished" podID="4fb42c05-fb07-4dae-975b-71d45dd52e03" containerID="60f4c049b4be2511e00ef200ba1ccf764c612a21c9eedb12330041b80adbe914" exitCode=255 Oct 30 17:52:15.209536 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.209274 2578 scope.go:122] "RemoveContainer" containerID="60f4c049b4be2511e00ef200ba1ccf764c612a21c9eedb12330041b80adbe914" Oct 30 17:52:15.209536 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.209275 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" event={"ID":"4fb42c05-fb07-4dae-975b-71d45dd52e03","Type":"ContainerDied","Data":"60f4c049b4be2511e00ef200ba1ccf764c612a21c9eedb12330041b80adbe914"} Oct 30 17:52:15.213805 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.211118 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vqltd" event={"ID":"d5d1adae-1728-499a-9d16-ec24f1d69160","Type":"ContainerStarted","Data":"06ec5fb6cb500a01567f6fcd99b57082fda5e62cb8a7adadbb3af9853bd1f97f"} Oct 30 17:52:15.221216 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.221072 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-x2wf2-g8lv5"] Oct 30 17:52:15.224288 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.224226 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" event={"ID":"e4aa5e79-72a0-4b20-a639-ac9b61be4759","Type":"ContainerStarted","Data":"67529b2281e22b886000bff73777f92cd758f9f25ba7e958b6c4922b51b90669"} Oct 30 17:52:15.240887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.240856 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-data-volume\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.240887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.240887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.241029 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.240906 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-crio-socket\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.241029 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.240922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.241111 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.241068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr2r4\" (UniqueName: \"kubernetes.io/projected/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-api-access-sr2r4\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.241550 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.241532 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:52:15.242421 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.242388 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-csr4p" event={"ID":"15f0fc8a-91c4-404a-91ec-b39914e83cae","Type":"ContainerStarted","Data":"b237fc030c991c4736cb3bfc76070dbde8658fa8c65ba4621c49e438e37429a2"} Oct 30 17:52:15.268356 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.268334 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-w7rbg\"" Oct 30 17:52:15.301295 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.301244 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" podStartSLOduration=389.896392755 podStartE2EDuration="6m33.301230731s" totalImagesPullingTime="3.404837976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:15.299694905 +0000 UTC m=+34.907389329" watchObservedRunningTime="2026-10-30 17:52:15.301230731 +0000 UTC m=+34.908925149" Oct 30 17:52:15.341403 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.341370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-crio-socket\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.341403 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.341402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.341544 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.341429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8v7l\" (UniqueName: \"kubernetes.io/projected/4ea379eb-5288-4f0f-90f4-712600a2fc01-kube-api-access-x8v7l\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.341544 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:15.341524 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:15.341640 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:15.341573 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm podName:9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa nodeName:}" failed. No retries permitted until 2026-10-30 17:52:15.841560021 +0000 UTC m=+35.449254436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sw2jb" (UID: "9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa") : configmap "kube-rbac-proxy" not found Oct 30 17:52:15.341680 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.341647 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-crio-socket\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.341753 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.341733 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2r4\" (UniqueName: \"kubernetes.io/projected/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-api-access-sr2r4\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.341918 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.341903 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-trusted-ca-bundle\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.341971 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.341946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-service-ca-bundle\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.342059 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.342045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ea379eb-5288-4f0f-90f4-712600a2fc01-archives-path\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.342167 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.342150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-data-volume\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.344463 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.342570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.344463 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.344418 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-data-volume\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.344463 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:15.344432 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:15.344657 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:15.344493 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls podName:9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa nodeName:}" failed. No retries permitted until 2026-10-30 17:52:15.844475847 +0000 UTC m=+35.452170265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sw2jb" (UID: "9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:15.369038 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.369005 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr2r4\" (UniqueName: \"kubernetes.io/projected/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-api-access-sr2r4\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.404837 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.404752 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t5n25" podStartSLOduration=1.564657635 podStartE2EDuration="5.40473758s" totalImagesPullingTime="3.840079945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:15.40402923 +0000 UTC m=+35.011723655" watchObservedRunningTime="2026-10-30 17:52:15.40473758 +0000 UTC m=+35.012432003" Oct 30 17:52:15.443554 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.443520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8v7l\" (UniqueName: \"kubernetes.io/projected/4ea379eb-5288-4f0f-90f4-712600a2fc01-kube-api-access-x8v7l\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.443679 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.443654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-trusted-ca-bundle\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.443712 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.443677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-service-ca-bundle\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.443776 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.443711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ea379eb-5288-4f0f-90f4-712600a2fc01-archives-path\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.444160 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.444136 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ea379eb-5288-4f0f-90f4-712600a2fc01-archives-path\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.444339 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.444319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-service-ca-bundle\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.444488 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.444469 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-trusted-ca-bundle\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.456333 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.456277 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-csr4p" podStartSLOduration=392.094229144 podStartE2EDuration="6m35.456261489s" totalImagesPullingTime="3.362032345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:15.455532836 +0000 UTC m=+35.063227260" watchObservedRunningTime="2026-10-30 17:52:15.456261489 +0000 UTC m=+35.063955914" Oct 30 17:52:15.458825 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.458803 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh"] Oct 30 17:52:15.462360 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:52:15.462333 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d610c80_f954_4f69_ab9a_e75c56c98652.slice/crio-3bf4f7b93de577cc28bf4cdd4b632aaf0262666f691543b4e172102314928297 WatchSource:0}: Error finding container 3bf4f7b93de577cc28bf4cdd4b632aaf0262666f691543b4e172102314928297: Status 404 returned error can't find the container with id 3bf4f7b93de577cc28bf4cdd4b632aaf0262666f691543b4e172102314928297 Oct 30 17:52:15.495658 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.495628 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8v7l\" (UniqueName: \"kubernetes.io/projected/4ea379eb-5288-4f0f-90f4-712600a2fc01-kube-api-access-x8v7l\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.543656 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.543625 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:15.847986 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.847954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.848407 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.848003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:15.848407 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:15.848054 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:15.848407 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:15.848120 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls podName:9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa nodeName:}" failed. No retries permitted until 2026-10-30 17:52:16.848105368 +0000 UTC m=+36.455799787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sw2jb" (UID: "9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:15.848407 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:15.848141 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:15.848407 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:15.848197 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm podName:9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa nodeName:}" failed. No retries permitted until 2026-10-30 17:52:16.848181442 +0000 UTC m=+36.455875882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sw2jb" (UID: "9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa") : configmap "kube-rbac-proxy" not found Oct 30 17:52:15.868032 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:15.867965 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-x2wf2-g8lv5"] Oct 30 17:52:15.872404 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:52:15.872377 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea379eb_5288_4f0f_90f4_712600a2fc01.slice/crio-afde3c89beda09c91adcc60ebd6d6823c774cdc8ae9d4a0911a77f5f21d8c65a WatchSource:0}: Error finding container afde3c89beda09c91adcc60ebd6d6823c774cdc8ae9d4a0911a77f5f21d8c65a: Status 404 returned error can't find the container with id afde3c89beda09c91adcc60ebd6d6823c774cdc8ae9d4a0911a77f5f21d8c65a Oct 30 17:52:16.192441 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.192365 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wql78"] Oct 30 17:52:16.210984 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.210953 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:16.223140 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.223116 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wql78"] Oct 30 17:52:16.230315 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.230290 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 17:52:16.230450 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.230389 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 17:52:16.230450 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.230441 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 17:52:16.230600 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.230450 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 17:52:16.234887 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.234867 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hzdpx\"" Oct 30 17:52:16.249742 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.249700 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/1.log" Oct 30 17:52:16.251192 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.251164 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/0.log" Oct 30 17:52:16.251305 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.251206 2578 generic.go:356] "Generic (PLEG): container finished" podID="4fb42c05-fb07-4dae-975b-71d45dd52e03" containerID="9d4cc54e9eac0995064b052a1728805f57581f4f07a54b60d85f9eddfac462f5" exitCode=255 Oct 30 17:52:16.251427 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.251399 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" event={"ID":"4fb42c05-fb07-4dae-975b-71d45dd52e03","Type":"ContainerDied","Data":"9d4cc54e9eac0995064b052a1728805f57581f4f07a54b60d85f9eddfac462f5"} Oct 30 17:52:16.251528 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.251461 2578 scope.go:122] "RemoveContainer" containerID="60f4c049b4be2511e00ef200ba1ccf764c612a21c9eedb12330041b80adbe914" Oct 30 17:52:16.251615 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.251602 2578 scope.go:122] "RemoveContainer" containerID="9d4cc54e9eac0995064b052a1728805f57581f4f07a54b60d85f9eddfac462f5" Oct 30 17:52:16.251834 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:16.251810 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5zcf9_openshift-console-operator(4fb42c05-fb07-4dae-975b-71d45dd52e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" podUID="4fb42c05-fb07-4dae-975b-71d45dd52e03" Oct 30 17:52:16.254464 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.254432 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" event={"ID":"4ea379eb-5288-4f0f-90f4-712600a2fc01","Type":"ContainerStarted","Data":"452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c"} Oct 30 17:52:16.254580 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.254477 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" event={"ID":"4ea379eb-5288-4f0f-90f4-712600a2fc01","Type":"ContainerStarted","Data":"afde3c89beda09c91adcc60ebd6d6823c774cdc8ae9d4a0911a77f5f21d8c65a"} Oct 30 17:52:16.256792 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.256771 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh" event={"ID":"3d610c80-f954-4f69-ab9a-e75c56c98652","Type":"ContainerStarted","Data":"3bf4f7b93de577cc28bf4cdd4b632aaf0262666f691543b4e172102314928297"} Oct 30 17:52:16.260075 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.260019 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fglxs" event={"ID":"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104","Type":"ContainerStarted","Data":"0c7d6cbe457d11143c85e47449fd7800aceaf613a7a8012fe59ca7b1299a4848"} Oct 30 17:52:16.354100 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.353414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-signing-cabundle\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:16.354100 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.353499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nslxb\" (UniqueName: \"kubernetes.io/projected/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-kube-api-access-nslxb\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:16.354100 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.353594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-signing-key\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:16.420944 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.420859 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" podStartSLOduration=1.420835538 podStartE2EDuration="1.420835538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:16.418265337 +0000 UTC m=+36.025959762" watchObservedRunningTime="2026-10-30 17:52:16.420835538 +0000 UTC m=+36.028529965" Oct 30 17:52:16.455025 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.454232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-signing-cabundle\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:16.455025 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.454308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nslxb\" (UniqueName: \"kubernetes.io/projected/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-kube-api-access-nslxb\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:16.455025 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.454377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-signing-key\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:16.455475 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.455425 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-signing-cabundle\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:16.460620 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.460570 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-signing-key\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:16.482761 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.482735 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nslxb\" (UniqueName: \"kubernetes.io/projected/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-kube-api-access-nslxb\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:16.534573 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.534533 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:16.858993 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.858951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:16.859490 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.859002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:16.859490 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:16.859099 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:16.859490 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:16.859117 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:16.859490 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:16.859162 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls podName:9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.859148201 +0000 UTC m=+38.466842620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sw2jb" (UID: "9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:16.859490 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:16.859177 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm podName:9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa nodeName:}" failed. No retries permitted until 2026-10-30 17:52:18.859170995 +0000 UTC m=+38.466865397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sw2jb" (UID: "9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa") : configmap "kube-rbac-proxy" not found Oct 30 17:52:16.866782 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:16.866756 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wql78"] Oct 30 17:52:16.870834 ip-10-0-130-250 kubenswrapper[2578]: W1030 17:52:16.870804 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45d20e6_c15c_4f25_9c13_d9cebc4775f0.slice/crio-617a558f570813a7e1e53a91fc54d214c8cec4ac686dd1a8a6459822de259180 WatchSource:0}: Error finding container 617a558f570813a7e1e53a91fc54d214c8cec4ac686dd1a8a6459822de259180: Status 404 returned error can't find the container with id 617a558f570813a7e1e53a91fc54d214c8cec4ac686dd1a8a6459822de259180 Oct 30 17:52:17.281705 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:17.281659 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh" event={"ID":"3d610c80-f954-4f69-ab9a-e75c56c98652","Type":"ContainerStarted","Data":"2e60c2b9b59806d97507b73d980e39769bc4b397524a6a29a1a134bfaa1fd294"} Oct 30 17:52:17.284284 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:17.284250 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" event={"ID":"e45d20e6-c15c-4f25-9c13-d9cebc4775f0","Type":"ContainerStarted","Data":"5043965b3a08a73f6128e5892807b8124d31607c529d550a7e415a75eca75eff"} Oct 30 17:52:17.284410 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:17.284292 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" event={"ID":"e45d20e6-c15c-4f25-9c13-d9cebc4775f0","Type":"ContainerStarted","Data":"617a558f570813a7e1e53a91fc54d214c8cec4ac686dd1a8a6459822de259180"} Oct 30 17:52:17.287245 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:17.287220 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/1.log" Oct 30 17:52:17.287598 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:17.287580 2578 scope.go:122] "RemoveContainer" containerID="9d4cc54e9eac0995064b052a1728805f57581f4f07a54b60d85f9eddfac462f5" Oct 30 17:52:17.287767 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:17.287747 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5zcf9_openshift-console-operator(4fb42c05-fb07-4dae-975b-71d45dd52e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" podUID="4fb42c05-fb07-4dae-975b-71d45dd52e03" Oct 30 17:52:17.329543 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:17.328213 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh" podStartSLOduration=1.641443406 podStartE2EDuration="3.328196258s" totalImagesPullingTime="1.686752852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:17.327649954 +0000 UTC m=+36.935344379" watchObservedRunningTime="2026-10-30 17:52:17.328196258 +0000 UTC m=+36.935890683" Oct 30 17:52:17.371682 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:17.371624 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" podStartSLOduration=1.371605689 podStartE2EDuration="1.371605689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:17.371000339 +0000 UTC m=+36.978694763" watchObservedRunningTime="2026-10-30 17:52:17.371605689 +0000 UTC m=+36.979300115" Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:18.070791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:18.070840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:18.070870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:18.070888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:18.070917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.071057 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.071107 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.071123 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:26.071102546 +0000 UTC m=+45.678796964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : secret "router-metrics-certs-default" not found Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.071144 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls podName:7cf0aaaf-7ca8-44b7-afad-829e4455e56a nodeName:}" failed. No retries permitted until 2026-10-30 17:52:26.071133908 +0000 UTC m=+45.678828313 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls") pod "dns-default-6fncr" (UID: "7cf0aaaf-7ca8-44b7-afad-829e4455e56a") : secret "dns-default-metrics-tls" not found Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.071057 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.071173 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls podName:1f8536c2-cd14-4d88-ab1d-63a2f032aeb5 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:26.071165553 +0000 UTC m=+45.678859957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r7rxt" (UID: "1f8536c2-cd14-4d88-ab1d-63a2f032aeb5") : secret "samples-operator-tls" not found Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.071231 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:26.071222284 +0000 UTC m=+45.678916699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : configmap references non-existent config key: service-ca.crt Oct 30 17:52:18.071452 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.071414 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:52:18.072423 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.071493 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls podName:867cb674-bd6b-4ca7-98a6-fa31bdaa132d nodeName:}" failed. No retries permitted until 2026-10-30 17:52:26.071474635 +0000 UTC m=+45.679169055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z2t26" (UID: "867cb674-bd6b-4ca7-98a6-fa31bdaa132d") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:52:18.878235 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:18.878201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:18.878440 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:18.878246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:18.878440 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.878325 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:52:18.878440 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.878371 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm podName:9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.878359491 +0000 UTC m=+42.486053906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sw2jb" (UID: "9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa") : configmap "kube-rbac-proxy" not found Oct 30 17:52:18.878765 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.878745 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:18.878879 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:18.878790 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls podName:9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.878778738 +0000 UTC m=+42.486473153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sw2jb" (UID: "9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:19.972442 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:19.972410 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xbxvk_97c8fb7c-e9a6-4eff-9a66-b70848c03a95/dns-node-resolver/0.log" Oct 30 17:52:20.303502 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:20.303416 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vqltd" event={"ID":"d5d1adae-1728-499a-9d16-ec24f1d69160","Type":"ContainerStarted","Data":"2d029bf9bb4cf371f5ffe62a0e1c1f5548616ad78c86e2572760d06122bd8a55"} Oct 30 17:52:20.303660 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:20.303643 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:20.349522 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:20.349457 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vqltd" podStartSLOduration=35.164398837 podStartE2EDuration="39.349432385s" totalImagesPullingTime="4.185033548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:20.348085534 +0000 UTC m=+39.955779972" watchObservedRunningTime="2026-10-30 17:52:20.349432385 +0000 UTC m=+39.957126808" Oct 30 17:52:20.552317 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:20.552275 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:20.552477 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:20.552342 2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:20.552681 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:20.552668 2578 scope.go:122] "RemoveContainer" containerID="9d4cc54e9eac0995064b052a1728805f57581f4f07a54b60d85f9eddfac462f5" Oct 30 17:52:20.552865 ip-10-0-130-250 kubenswrapper[2578]: E1030 17:52:20.552847 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5zcf9_openshift-console-operator(4fb42c05-fb07-4dae-975b-71d45dd52e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" podUID="4fb42c05-fb07-4dae-975b-71d45dd52e03" Oct 30 17:52:20.769258 ip-10-0-130-250 kubenswrapper[2578]: I1030 17:52:20.769236 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7gt8q_d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0/node-ca/0.log" Oct 30 17:52:20.987136 ip-10-0-130-250 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 17:52:21.002343 ip-10-0-130-250 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 17:52:21.002521 ip-10-0-130-250 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 17:52:21.002575 ip-10-0-130-250 systemd[1]: kubelet.service: Consumed 2.866s CPU time. Oct 30 17:52:21.011624 ip-10-0-130-250 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:52:21.072667 ip-10-0-130-250 kubenswrapper[4883]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:52:21.072667 ip-10-0-130-250 kubenswrapper[4883]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:52:21.072667 ip-10-0-130-250 kubenswrapper[4883]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:52:21.072667 ip-10-0-130-250 kubenswrapper[4883]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:52:21.074801 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074715 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:21.074801 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074801 4883 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074805 4883 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074808 4883 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074811 4883 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074814 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074817 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074819 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074823 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074825 4883 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074828 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074831 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074833 4883 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074836 4883 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074843 4883 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074846 4883 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074848 4883 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074851 4883 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074853 4883 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074856 4883 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074859 4883 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:21.074861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074862 4883 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074865 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074868 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074871 4883 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074874 4883 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074877 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074879 4883 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074882 4883 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074885 4883 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074887 4883 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074890 4883 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074892 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074895 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074897 4883 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074900 4883 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074902 4883 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074905 4883 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074908 4883 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074910 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:21.075386 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074915 4883 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074919 4883 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074922 4883 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074926 4883 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074930 4883 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074953 4883 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074956 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074959 4883 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074962 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074964 4883 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074967 4883 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074970 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074973 4883 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074975 4883 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074978 4883 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074981 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074984 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074987 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074989 4883 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:21.075870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074993 4883 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074996 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.074998 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075001 4883 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075004 4883 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075007 4883 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075009 4883 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075012 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075014 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075017 4883 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075021 4883 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075023 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075026 4883 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075029 4883 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075031 4883 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075034 4883 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075037 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075039 4883 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075042 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075045 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:21.076356 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075047 4883 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075050 4883 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075053 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075055 4883 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075058 4883 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075061 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075063 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075067 4883 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075070 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075072 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075075 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075077 4883 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075080 4883 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075083 4883 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075085 4883 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075088 4883 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075091 4883 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075094 4883 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075096 4883 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075099 4883 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:21.076867 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075102 4883 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075104 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075107 4883 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075111 4883 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075113 4883 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075116 4883 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075118 4883 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075902 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075911 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075914 4883 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075917 4883 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075920 4883 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075923 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075927 4883 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075930 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075946 4883 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075949 4883 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075952 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075955 4883 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:21.077399 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075959 4883 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075961 4883 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075964 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075967 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075969 4883 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075975 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075978 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075980 4883 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075983 4883 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075986 4883 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075988 4883 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075991 4883 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075994 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075996 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.075999 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076002 4883 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076004 4883 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076007 4883 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076009 4883 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076012 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:21.077913 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076014 4883 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076017 4883 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076020 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076022 4883 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076025 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076027 4883 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076030 4883 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076033 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076036 4883 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076038 4883 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076041 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076043 4883 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076046 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076051 4883 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076055 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076058 4883 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076061 4883 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076064 4883 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076067 4883 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:21.078429 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076070 4883 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076072 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076075 4883 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076077 4883 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076080 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076083 4883 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076086 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076088 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076091 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076093 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076096 4883 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076098 4883 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076101 4883 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076104 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076106 4883 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076109 4883 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076111 4883 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076113 4883 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076116 4883 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:21.079050 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076118 4883 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076121 4883 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076124 4883 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076126 4883 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076129 4883 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076132 4883 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076135 4883 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076137 4883 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076140 4883 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076142 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076145 4883 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076148 4883 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076151 4883 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076154 4883 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076156 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076159 4883 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076162 4883 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076166 4883 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076169 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076172 4883 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:21.079530 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076175 4883 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076178 4883 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076181 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076183 4883 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076186 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076188 4883 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076191 4883 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076193 4883 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076196 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076198 4883 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076201 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076203 4883 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076206 4883 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076209 4883 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076212 4883 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076214 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076699 4883 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076704 4883 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076710 4883 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076713 4883 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:21.080052 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076716 4883 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076718 4883 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076721 4883 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076725 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076727 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076730 4883 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076732 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076736 4883 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076741 4883 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076744 4883 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076746 4883 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076749 4883 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076752 4883 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076754 4883 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076757 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076759 4883 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076762 4883 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076764 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076767 4883 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:21.080549 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076769 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076772 4883 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076775 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076778 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076781 4883 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076785 4883 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076788 4883 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076791 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076793 4883 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076796 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076799 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076803 4883 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076805 4883 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076808 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076810 4883 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076820 4883 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076825 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076827 4883 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076830 4883 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:21.081035 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076833 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076835 4883 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076838 4883 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076841 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076843 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076846 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076848 4883 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076851 4883 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076854 4883 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076856 4883 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076859 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076862 4883 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076865 4883 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076867 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076870 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076873 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076876 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076878 4883 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076881 4883 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076884 4883 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:21.081512 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076886 4883 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076889 4883 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076891 4883 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076895 4883 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076899 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076901 4883 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076904 4883 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076906 4883 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076909 4883 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076911 4883 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076914 4883 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076917 4883 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076919 4883 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076922 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076924 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076927 4883 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076929 4883 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076944 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076947 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076950 4883 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:21.082037 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076953 4883 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076955 4883 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076958 4883 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076961 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076964 4883 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076966 4883 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076969 4883 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076971 4883 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076974 4883 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076976 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076979 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076982 4883 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076984 4883 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076988 4883 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076990 4883 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076993 4883 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.076997 4883 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077000 4883 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077003 4883 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:21.082563 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077005 4883 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077008 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077011 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077013 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077016 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077116 4883 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077135 4883 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077144 4883 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077150 4883 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077154 4883 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077158 4883 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077162 4883 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077169 4883 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077172 4883 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077176 4883 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077180 4883 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077184 4883 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077187 4883 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077190 4883 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077193 4883 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077196 4883 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077199 4883 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077202 4883 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:52:21.083316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077207 4883 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077210 4883 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077213 4883 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077217 4883 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077220 4883 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077224 4883 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077228 4883 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077232 4883 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077235 4883 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077238 4883 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077242 4883 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077244 4883 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077248 4883 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077251 4883 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077255 4883 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077258 4883 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077261 4883 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077264 4883 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077267 4883 flags.go:64] FLAG: --enable-server="true" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077270 4883 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077275 4883 flags.go:64] FLAG: --event-burst="100" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077280 4883 flags.go:64] FLAG: --event-qps="50" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077283 4883 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077286 4883 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:52:21.084120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077289 4883 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077294 4883 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077297 4883 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077300 4883 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077303 4883 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077306 4883 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077309 4883 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077312 4883 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077315 4883 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077318 4883 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077321 4883 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077324 4883 flags.go:64] FLAG: --feature-gates="" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077328 4883 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077331 4883 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077336 4883 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077339 4883 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077347 4883 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077350 4883 flags.go:64] FLAG: --help="false" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077353 4883 flags.go:64] FLAG: --hostname-override="ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077356 4883 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077360 4883 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077363 4883 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077366 4883 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:52:21.084708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077369 4883 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077372 4883 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077375 4883 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077378 4883 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077381 4883 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077384 4883 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077389 4883 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077392 4883 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077395 4883 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077398 4883 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077401 4883 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077404 4883 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077407 4883 flags.go:64] FLAG: --lock-file="" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077410 4883 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077413 4883 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077416 4883 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077422 4883 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077425 4883 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077428 4883 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077431 4883 flags.go:64] FLAG: --logging-format="text" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077434 4883 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077438 4883 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077440 4883 flags.go:64] FLAG: --manifest-url="" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077444 4883 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077449 4883 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:52:21.085343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077452 4883 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077456 4883 flags.go:64] FLAG: --max-pods="110" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077459 4883 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077462 4883 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077465 4883 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077468 4883 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077471 4883 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077474 4883 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077477 4883 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077485 4883 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077488 4883 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077491 4883 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077494 4883 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077499 4883 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077502 4883 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077505 4883 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077508 4883 flags.go:64] FLAG: --port="10250" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077511 4883 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077515 4883 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08935549a12d46347" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077518 4883 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077521 4883 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077524 4883 flags.go:64] FLAG: --register-node="true" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077527 4883 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077531 4883 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077534 4883 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077537 4883 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:52:21.085983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077540 4883 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077544 4883 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077547 4883 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077550 4883 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077553 4883 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077557 4883 flags.go:64] FLAG: --runonce="false" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077560 4883 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077564 4883 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077567 4883 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077570 4883 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077574 4883 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077577 4883 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077580 4883 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077583 4883 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077586 4883 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077589 4883 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077592 4883 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077595 4883 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077599 4883 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077604 4883 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077607 4883 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077612 4883 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077615 4883 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077618 4883 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077623 4883 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:52:21.086682 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077626 4883 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077629 4883 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077632 4883 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077635 4883 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077638 4883 flags.go:64] FLAG: --v="2" Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077643 4883 flags.go:64] FLAG: --version="false" Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077647 4883 flags.go:64] FLAG: --vmodule="" Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077651 4883 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.077655 4883 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077788 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077792 4883 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077795 4883 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077798 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077801 4883 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077803 4883 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077806 4883 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077809 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077812 4883 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077814 4883 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077817 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077819 4883 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:21.087337 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077822 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077824 4883 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077827 4883 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077830 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077832 4883 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077837 4883 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077839 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077842 4883 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077844 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077847 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077850 4883 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077852 4883 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077856 4883 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077860 4883 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077865 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077869 4883 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077875 4883 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077881 4883 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077884 4883 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:21.087891 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077887 4883 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077889 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077892 4883 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077895 4883 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077898 4883 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077900 4883 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077903 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077906 4883 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077909 4883 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077912 4883 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077915 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077918 4883 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077920 4883 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077923 4883 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077928 4883 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077944 4883 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077947 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077950 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077954 4883 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:21.088385 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077957 4883 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077960 4883 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077963 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077965 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077968 4883 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077971 4883 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077974 4883 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077976 4883 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077979 4883 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077982 4883 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077985 4883 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077987 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077990 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077993 4883 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.077995 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078000 4883 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078003 4883 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078005 4883 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078008 4883 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078011 4883 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:21.088861 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078013 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078016 4883 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078019 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078025 4883 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078028 4883 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078031 4883 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078033 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078036 4883 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078039 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078041 4883 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078044 4883 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078048 4883 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078050 4883 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078053 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078070 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078075 4883 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078079 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078082 4883 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078085 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078088 4883 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:21.089373 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078091 4883 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078094 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078096 4883 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078099 4883 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078101 4883 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078104 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078107 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078111 4883 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078114 4883 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078116 4883 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078119 4883 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078122 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078124 4883 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078127 4883 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078129 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078132 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:21.089870 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.078160 4883 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078197 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078201 4883 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078204 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078207 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078210 4883 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078213 4883 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078215 4883 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078219 4883 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078222 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078225 4883 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078229 4883 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078233 4883 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078236 4883 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078239 4883 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078241 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078244 4883 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078246 4883 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078249 4883 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:21.090280 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078252 4883 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078254 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078257 4883 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078260 4883 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078264 4883 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078267 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078270 4883 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078273 4883 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078275 4883 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078279 4883 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078283 4883 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078286 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078289 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078291 4883 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078295 4883 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078297 4883 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078300 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078303 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078305 4883 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:21.090783 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078307 4883 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078310 4883 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078314 4883 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078317 4883 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078319 4883 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078322 4883 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078325 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078327 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078330 4883 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078332 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078335 4883 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078338 4883 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078340 4883 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078343 4883 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078345 4883 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078348 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078350 4883 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078354 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078357 4883 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:21.091273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078359 4883 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078362 4883 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078364 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078367 4883 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078370 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078372 4883 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078375 4883 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078377 4883 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078380 4883 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078383 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078385 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078388 4883 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078390 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078393 4883 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078395 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078401 4883 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078403 4883 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078406 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078409 4883 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078411 4883 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:21.091759 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078414 4883 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078417 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078419 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078422 4883 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078424 4883 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078427 4883 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078430 4883 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078432 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078435 4883 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078438 4883 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078442 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078445 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078449 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078451 4883 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078454 4883 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078456 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078459 4883 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078462 4883 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078464 4883 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:21.092282 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078467 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078469 4883 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078471 4883 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078475 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078477 4883 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078480 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078482 4883 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078485 4883 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078488 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078491 4883 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.078494 4883 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.083327 4883 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.083349 4883 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.083374 4883 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.083381 4883 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083609 4883 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083615 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083619 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083651 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083722 4883 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:21.092746 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083729 4883 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083732 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083748 4883 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083751 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083861 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083866 4883 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083871 4883 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083874 4883 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083877 4883 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083880 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083883 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083885 4883 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083888 4883 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083891 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083894 4883 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083896 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083899 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083902 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083905 4883 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083908 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:21.093439 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083910 4883 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083913 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083918 4883 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083921 4883 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083924 4883 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083926 4883 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083929 4883 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083958 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083961 4883 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083964 4883 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083968 4883 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083971 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083974 4883 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083977 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083980 4883 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083983 4883 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083986 4883 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083988 4883 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083991 4883 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083994 4883 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:21.093943 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.083996 4883 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084002 4883 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084004 4883 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084007 4883 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084010 4883 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084013 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084016 4883 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084018 4883 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084021 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084024 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084026 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084029 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084032 4883 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084034 4883 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084037 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084040 4883 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084043 4883 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084047 4883 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084053 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:21.094419 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084056 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084060 4883 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084064 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084067 4883 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084071 4883 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084074 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084078 4883 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084081 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084086 4883 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084089 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084096 4883 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084100 4883 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084103 4883 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084107 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084110 4883 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084113 4883 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084116 4883 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084119 4883 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084122 4883 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:21.094875 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084125 4883 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084128 4883 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084131 4883 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084134 4883 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084137 4883 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084140 4883 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084143 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084145 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084148 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084151 4883 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084153 4883 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084156 4883 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084158 4883 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084161 4883 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084164 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084167 4883 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084170 4883 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084173 4883 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084175 4883 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084178 4883 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:21.095361 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084181 4883 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084183 4883 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084186 4883 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.084215 4883 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084384 4883 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084390 4883 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084393 4883 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084396 4883 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084398 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084402 4883 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084405 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084407 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084410 4883 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084413 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084416 4883 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084419 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:21.095910 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084421 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084424 4883 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084426 4883 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084429 4883 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084432 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084435 4883 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084437 4883 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084440 4883 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084442 4883 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084445 4883 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084448 4883 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084451 4883 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084454 4883 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084456 4883 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084459 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084462 4883 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084464 4883 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084467 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084469 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:21.096325 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084472 4883 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084475 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084477 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084480 4883 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084483 4883 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084485 4883 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084488 4883 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084491 4883 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084493 4883 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084496 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084499 4883 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084503 4883 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084506 4883 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084509 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084511 4883 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084514 4883 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084517 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084519 4883 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084522 4883 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:21.096821 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084524 4883 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084527 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084529 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084532 4883 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084534 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084537 4883 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084540 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084542 4883 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084545 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084548 4883 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084550 4883 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084553 4883 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084555 4883 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084558 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084560 4883 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084563 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084565 4883 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084568 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084570 4883 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084573 4883 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:21.097309 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084575 4883 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084579 4883 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084581 4883 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084584 4883 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084586 4883 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084589 4883 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084591 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084594 4883 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084596 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084599 4883 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084602 4883 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084604 4883 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084607 4883 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084611 4883 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084615 4883 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084618 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084621 4883 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084623 4883 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084626 4883 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084629 4883 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:21.097808 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084631 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084634 4883 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084636 4883 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084638 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084641 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084643 4883 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084646 4883 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084649 4883 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084651 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084654 4883 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084656 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084659 4883 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084661 4883 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084664 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084670 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084673 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:21.098314 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.084696 4883 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084728 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084732 4883 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084735 4883 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084737 4883 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084740 4883 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084743 4883 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084746 4883 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084749 4883 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084751 4883 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084754 4883 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084757 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084759 4883 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084762 4883 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084765 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084768 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084770 4883 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084773 4883 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084775 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:52:21.098703 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084778 4883 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084781 4883 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084783 4883 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084786 4883 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084788 4883 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084791 4883 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084793 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084796 4883 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084800 4883 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084803 4883 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084806 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084809 4883 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084821 4883 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084824 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084827 4883 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084830 4883 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084833 4883 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084835 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084838 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084841 4883 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:52:21.099183 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084844 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084846 4883 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084849 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084851 4883 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084854 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084856 4883 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084859 4883 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084862 4883 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084864 4883 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084867 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084870 4883 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084872 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084875 4883 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084877 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084880 4883 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084882 4883 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084884 4883 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084887 4883 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084889 4883 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084892 4883 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:52:21.099662 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084894 4883 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084897 4883 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084900 4883 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084903 4883 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084905 4883 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084908 4883 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084911 4883 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084915 4883 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084917 4883 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084920 4883 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084924 4883 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084928 4883 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084945 4883 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084950 4883 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084954 4883 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084957 4883 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084960 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084962 4883 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084965 4883 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:52:21.100153 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084968 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084971 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084973 4883 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084976 4883 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084979 4883 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084981 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084984 4883 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084986 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084989 4883 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084992 4883 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084995 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.084998 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085000 4883 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085003 4883 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085006 4883 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085008 4883 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085011 4883 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085014 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085016 4883 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085020 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:52:21.100613 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085023 4883 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085025 4883 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085028 4883 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085030 4883 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085032 4883 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085035 4883 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085038 4883 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085040 4883 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:21.085044 4883 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.085108 4883 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.086108 4883 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.086169 4883 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.086347 4883 server.go:1014] "Starting client certificate rotation" Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.086488 4883 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.086529 4883 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:46:40 +0000 UTC" deadline="2028-04-04 04:44:56.419995346 +0000 UTC" Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.086558 4883 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12514h52m35.333439551s" Oct 30 17:52:21.101273 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.086806 4883 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:52:21.107222 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.107190 4883 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:52:21.108778 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.108762 4883 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:52:21.110618 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.110603 4883 log.go:25] "Validated CRI v1 image API" Oct 30 17:52:21.111326 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.111308 4883 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:52:21.113205 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.113184 4883 fs.go:135] Filesystem UUIDs: map[1fb7ba10-e3a7-4048-b76f-ea246cdc9860:/dev/nvme0n1p4 2288559e-90c2-49ee-98cf-0e4c6ee82ca7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 17:52:21.113440 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.113205 4883 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06ec5fb6cb500a01567f6fcd99b57082fda5e62cb8a7adadbb3af9853bd1f97f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06ec5fb6cb500a01567f6fcd99b57082fda5e62cb8a7adadbb3af9853bd1f97f/userdata/shm major:0 minor:283 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0825799559f49d5f45e640d862efec0735c7504f44a3954d773d8e6f8225d9e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0825799559f49d5f45e640d862efec0735c7504f44a3954d773d8e6f8225d9e3/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c7d6cbe457d11143c85e47449fd7800aceaf613a7a8012fe59ca7b1299a4848/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c7d6cbe457d11143c85e47449fd7800aceaf613a7a8012fe59ca7b1299a4848/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0cd0faf02ac1d0545ae1275e67b860caa84ccb64b1c0455eab979dbaad30084b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0cd0faf02ac1d0545ae1275e67b860caa84ccb64b1c0455eab979dbaad30084b/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d11ff40ffeed3a7b885575145b1a5889d89641ddf5c5dc4bde2ebf43e652912/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d11ff40ffeed3a7b885575145b1a5889d89641ddf5c5dc4bde2ebf43e652912/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19cd058eb83329fa6bc4277bf0e5e8faa72e400ef58a8faa61470589005876c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19cd058eb83329fa6bc4277bf0e5e8faa72e400ef58a8faa61470589005876c9/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a672c72572bbbc22e25fbf1853e3e557e72d76e81375d5db4e9e71412af9561/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a672c72572bbbc22e25fbf1853e3e557e72d76e81375d5db4e9e71412af9561/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2cdaadf0279074953b6cff64d8032e4817e8b7d1b9e297ce3e09dbab274dd9a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2cdaadf0279074953b6cff64d8032e4817e8b7d1b9e297ce3e09dbab274dd9a0/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f5d511d548c2fdefd45d8bdd0b8065dfcfacab0f31ac1c49fd7a36f41b150e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f5d511d548c2fdefd45d8bdd0b8065dfcfacab0f31ac1c49fd7a36f41b150e3/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3866e6ffabee47730af52b48828cedea8224c425c2f6bcbf19d40c024946200e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3866e6ffabee47730af52b48828cedea8224c425c2f6bcbf19d40c024946200e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3952840a381d0ce2f4744ec7477a9cf0c6c5fae507720774505f073f2d2e8bbb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3952840a381d0ce2f4744ec7477a9cf0c6c5fae507720774505f073f2d2e8bbb/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a9bbea95be20bac1e1bb2c2164a2994870d7dee69975397b9cdf1131560a066/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a9bbea95be20bac1e1bb2c2164a2994870d7dee69975397b9cdf1131560a066/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3bf4f7b93de577cc28bf4cdd4b632aaf0262666f691543b4e172102314928297/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3bf4f7b93de577cc28bf4cdd4b632aaf0262666f691543b4e172102314928297/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/617a558f570813a7e1e53a91fc54d214c8cec4ac686dd1a8a6459822de259180/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/617a558f570813a7e1e53a91fc54d214c8cec4ac686dd1a8a6459822de259180/userdata/shm major:0 minor:309 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f8dee39d1523e2e2efed08471bf46398d73d72028d5b0edae754cac4eb95746/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f8dee39d1523e2e2efed08471bf46398d73d72028d5b0edae754cac4eb95746/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afde3c89beda09c91adcc60ebd6d6823c774cdc8ae9d4a0911a77f5f21d8c65a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afde3c89beda09c91adcc60ebd6d6823c774cdc8ae9d4a0911a77f5f21d8c65a/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3f27e6dc43e7301384b86d9bda5ce54b9cea35fe48942ba6912fffa683f275b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3f27e6dc43e7301384b86d9bda5ce54b9cea35fe48942ba6912fffa683f275b/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8f683348012bd6873a25ecaf7c515e4200ff75713d9f8105822f8efbec1d83f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8f683348012bd6873a25ecaf7c515e4200ff75713d9f8105822f8efbec1d83f/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfd8e5655731d2860f1ddab8ce6a6ddb225411b8348500bbf21f6e92f55f1903/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfd8e5655731d2860f1ddab8ce6a6ddb225411b8348500bbf21f6e92f55f1903/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9f90e7f00f9665d52b7a1c5f5b916dbc31411be8d913d761cf750fc3e6f3ede/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9f90e7f00f9665d52b7a1c5f5b916dbc31411be8d913d761cf750fc3e6f3ede/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed98329f6d9d0ac3f0bb6062d734f0d64e2e785e26f672905bb7532f1de9b0f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed98329f6d9d0ac3f0bb6062d734f0d64e2e785e26f672905bb7532f1de9b0f8/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff8e2d857292e8512cc53593b73ef901f6c336dc2f9a9b24e074830970341052/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff8e2d857292e8512cc53593b73ef901f6c336dc2f9a9b24e074830970341052/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06498a3b-c75b-4bf5-9fc7-beced86dbfc5/volumes/kubernetes.io~projected/kube-api-access-gtlqq:{mountpoint:/var/lib/kubelet/pods/06498a3b-c75b-4bf5-9fc7-beced86dbfc5/volumes/kubernetes.io~projected/kube-api-access-gtlqq major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07abda71-7d42-43e6-9878-90ab7da7d9c4/volumes/kubernetes.io~projected/kube-api-access-xlkgn:{mountpoint:/var/lib/kubelet/pods/07abda71-7d42-43e6-9878-90ab7da7d9c4/volumes/kubernetes.io~projected/kube-api-access-xlkgn major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15f0fc8a-91c4-404a-91ec-b39914e83cae/volumes/kubernetes.io~projected/kube-api-access-w9756:{mountpoint:/var/lib/kubelet/pods/15f0fc8a-91c4-404a-91ec-b39914e83cae/volumes/kubernetes.io~projected/kube-api-access-w9756 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15f0fc8a-91c4-404a-91ec-b39914e83cae/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/15f0fc8a-91c4-404a-91ec-b39914e83cae/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19bd958f-7d20-4da4-818f-06e23e01ab37/volumes/kubernetes.io~projected/kube-api-access-rtx8j:{mountpoint:/var/lib/kubelet/pods/19bd958f-7d20-4da4-818f-06e23e01ab37/volumes/kubernetes.io~projected/kube-api-access-rtx8j major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5/volumes/kubernetes.io~projected/kube-api-access-pmtbq:{mountpoint:/var/lib/kubelet/pods/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5/volumes/kubernetes.io~projected/kube-api-access-pmtbq major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ce04018-ba31-464c-8996-967b9100f174/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2ce04018-ba31-464c-8996-967b9100f174/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d610c80-f954-4f69-ab9a-e75c56c98652/volumes/kubernetes.io~projected/kube-api-access-gv255:{mountpoint:/var/lib/kubelet/pods/3d610c80-f954-4f69-ab9a-e75c56c98652/volumes/kubernetes.io~projected/kube-api-access-gv255 major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4260cb2f-efee-4b45-9f73-a48757f3ce75/volumes/kubernetes.io~projected/kube-api-access-zr6bn:{mountpoint:/var/lib/kubelet/pods/4260cb2f-efee-4b45-9f73-a48757f3ce75/volumes/kubernetes.io~projected/kube-api-access-zr6bn major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4260cb2f-efee-4b45-9f73-a48757f3ce75/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/4260cb2f-efee-4b45-9f73-a48757f3ce75/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4260cb2f-efee-4b45-9f73-a48757f3ce75/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/4260cb2f-efee-4b45-9f73-a48757f3ce75/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea379eb-5288-4f0f-90f4-712600a2fc01/volumes/kubernetes.io~projected/kube-api-access-x8v7l:{mountpoint:/var/lib/kubelet/pods/4ea379eb-5288-4f0f-90f4-712600a2fc01/volumes/kubernetes.io~projected/kube-api-access-x8v7l major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fb42c05-fb07-4dae-975b-71d45dd52e03/volumes/kubernetes.io~projected/kube-api-access-67r2z:{mountpoint:/var/lib/kubelet/pods/4fb42c05-fb07-4dae-975b-71d45dd52e03/volumes/kubernetes.io~projected/kube-api-access-67r2z major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fb42c05-fb07-4dae-975b-71d45dd52e03/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4fb42c05-fb07-4dae-975b-71d45dd52e03/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e94fb26-859a-48d2-a493-200543aa1881/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5e94fb26-859a-48d2-a493-200543aa1881/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e94fb26-859a-48d2-a493-200543aa1881/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5e94fb26-859a-48d2-a493-200543aa1881/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e94fb26-859a-48d2-a493-200543aa1881/volumes/kubernetes.io~projected/kube-api-access-vmnr8:{mountpoint:/var/lib/kubelet/pods/5e94fb26-859a-48d2-a493-200543aa1881/volumes/kubernetes.io~projected/kube-api-access-vmnr8 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cf0aaaf-7ca8-44b7-afad-829e4455e56a/volumes/kubernetes.io~projected/kube-api-access-88w5h:{mountpoint:/var/lib/kubelet/pods/7cf0aaaf-7ca8-44b7-afad-829e4455e56a/volumes/kubernetes.io~projected/kube-api-access-88w5h major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881/volumes/kubernetes.io~projected/kube-api-access-r6kzk:{mountpoint:/var/lib/kubelet/pods/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881/volumes/kubernetes.io~projected/kube-api-access-r6kzk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/867cb674-bd6b-4ca7-98a6-fa31bdaa132d/volumes/kubernetes.io~projected/kube-api-access-bwdl6:{mountpoint:/var/lib/kubelet/pods/867cb674-bd6b-4ca7-98a6-fa31bdaa132d/volumes/kubernetes.io~projected/kube-api-access-bwdl6 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97c8fb7c-e9a6-4eff-9a66-b70848c03a95/volumes/kubernetes.io~projected/kube-api-access-plhvw:{mountpoint:/var/lib/kubelet/pods/97c8fb7c-e9a6-4eff-9a66-b70848c03a95/volumes/kubernetes.io~projected/kube-api-access-plhvw major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa/volumes/kubernetes.io~projected/kube-api-access-sr2r4:{mountpoint:/var/lib/kubelet/pods/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa/volumes/kubernetes.io~projected/kube-api-access-sr2r4 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d6f83f6-600c-4fa7-a20b-6a179c69bffc/volumes/kubernetes.io~projected/kube-api-access-j6mwh:{mountpoint:/var/lib/kubelet/pods/9d6f83f6-600c-4fa7-a20b-6a179c69bffc/volumes/kubernetes.io~projected/kube-api-access-j6mwh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6956a31-29a1-4cd7-bc8b-9cda43dc2e73/volumes/kubernetes.io~projected/kube-api-access-4xx66:{mountpoint:/var/lib/kubelet/pods/b6956a31-29a1-4cd7-bc8b-9cda43dc2e73/volumes/kubernetes.io~projected/kube-api-access-4xx66 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba65a282-0992-49ce-b4bc-108c8f69016d/volumes/kubernetes.io~projected/kube-api-access-2dxbs:{mountpoint:/var/lib/kubelet/pods/ba65a282-0992-49ce-b4bc-108c8f69016d/volumes/kubernetes.io~projected/kube-api-access-2dxbs major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1aa6b58-d1c3-40a0-873c-86b44c438fac/volumes/kubernetes.io~projected/kube-api-access-j9pq9:{mountpoint:/var/lib/kubelet/pods/c1aa6b58-d1c3-40a0-873c-86b44c438fac/volumes/kubernetes.io~projected/kube-api-access-j9pq9 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1aa6b58-d1c3-40a0-873c-86b44c438fac/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c1aa6b58-d1c3-40a0-873c-86b44c438fac/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1e25bb5-1c5b-449e-a7f0-0deb010035ed/volumes/kubernetes.io~projected/kube-api-access-h54zm:{mountpoint:/var/lib/kubelet/pods/d1e25bb5-1c5b-449e-a7f0-0deb010035ed/volumes/kubernetes.io~projected/kube-api-access-h54zm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0/volumes/kubernetes.io~projected/kube-api-access-mgj9b:{mountpoint:/var/lib/kubelet/pods/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0/volumes/kubernetes.io~projected/kube-api-access-mgj9b major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4c590f5-af8b-4994-8b03-7f5cee045767/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d4c590f5-af8b-4994-8b03-7f5cee045767/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4c590f5-af8b-4994-8b03-7f5cee045767/volumes/kubernetes.io~projected/kube-api-access-692s4:{mountpoint:/var/lib/kubelet/pods/d4c590f5-af8b-4994-8b03-7f5cee045767/volumes/kubernetes.io~projected/kube-api-access-692s4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4c590f5-af8b-4994-8b03-7f5cee045767/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d4c590f5-af8b-4994-8b03-7f5cee045767/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5d1adae-1728-499a-9d16-ec24f1d69160/volumes/kubernetes.io~projected/kube-api-access-4s2j6:{mountpoint:/var/lib/kubelet/pods/d5d1adae-1728-499a-9d16-ec24f1d69160/volumes/kubernetes.io~projected/kube-api-access-4s2j6 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e45d20e6-c15c-4f25-9c13-d9cebc4775f0/volumes/kubernetes.io~projected/kube-api-access-nslxb:{mountpoint:/var/lib/kubelet/pods/e45d20e6-c15c-4f25-9c13-d9cebc4775f0/volumes/kubernetes.io~projected/kube-api-access-nslxb major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e45d20e6-c15c-4f25-9c13-d9cebc4775f0/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/e45d20e6-c15c-4f25-9c13-d9cebc4775f0/volumes/kubernetes.io~secret/signing-key major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4aa5e79-72a0-4b20-a639-ac9b61be4759/volumes/kubernetes.io~projected/kube-api-access-nf98n:{mountpoint:/var/lib/kubelet/pods/e4aa5e79-72a0-4b20-a639-ac9b61be4759/volumes/kubernetes.io~projected/kube-api-access-nf98n major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4aa5e79-72a0-4b20-a639-ac9b61be4759/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e4aa5e79-72a0-4b20-a639-ac9b61be4759/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/988a82fd52c276131f40ff6cb9c390c2047ae5dcd4cbff3303a1ce1a9538901d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c7d1b0f98470f0bc8100ca4fa02e2d208fbd21a136e84d6dbe47e305ecf10182/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b7b6e24529a76ad0877e95c4b90f28fc171b415f7f67fa1c540c695c8c760d44/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/360d25179fdde73203737be0ea338dc901ba331e671fcce62924ee78bb1528ec/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/834ce35de8fb41bb1e2ad7dee80e90d94f0be881328493e70aeef23b8369950a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/95cf180b223d42049024564dbed0478b9d4c2ef250d305bfd664a59fa440cbcf/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8e0e58d4ba2f62e4d80f911a5bbfff28de9a13a493854c761a6f687db9be7776/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b8e1120a83ee23988c4d8cdb4f6f464ab2a6de05bbab454e92e4805f7993ccfb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/47272fd11d29915f0f3b055a79299d77f51495029c96fa9c646c3b8357b5fc18/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d76d919739916e581135c8162da61552d484fe89439a17918d6a5ac119822418/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/fd990a543b757d24a60ee98c28f0a695c00471012bb438a4abd6d2339de397c0/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/8455f6fd3cdb9ee7d685cebdae63475872bde88b7901962c38a701eecaabae2e/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/69c274f1418236ed3b250243a07165712098dbedd0e8660fc403a9bc234b874c/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/fb91fd0627111159a917a1a5a83e37f6f4ce31a974329f9dda95141290f5d8e1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0ef4b134e0d7429156f4213a463c427ff1d20813aad759c63256fdd5b73be78b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/8c2c691a99f3f5007835c210f658bb12192a360f6e9d0b99b1ca6dc877a8aefd/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e7d74c5f5fd7edd901986e7bd0aac17535fe590505738f5a021ea84376c5652e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0bb40dc882d17e44ad3cf8e508660bc50d590b7923ec93c0e50ddde13de853d8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ba00589760b5b6f33224a81682158a5372e1a2be9a1604a83b2de1929acadfde/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b0343a56901c78f7e65e72a18e7e79cc866f9ce52f1ed17d56de5ed7df7fa03b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d5b6a90d7303c7145c1cc9fe7f30863cb5de43f6ec7af05eb4d00b2b62f484a7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/4bdf85c8e889b1682085f837e13e7387954839b3a7fbe0b678d6ec386f7cb8d1/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/e25bad9405538eb629b5be33ceb6455e8866c7b8af4b1681c6da52e8019d58c6/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/85d628eeaf12daedd836ef508d3786c4fb8c8208fe878fd3771e8343506d78d3/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/e46dafaa6b62f061e0b68e5bbad241f08637675e015a221cb36115fdeb08af41/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/c42979fdbf2e6cb7527ad11aac40fe5823ff491c05b53459b039be21f36f1be8/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/f2c29792c6d2e2219170fec07a24ff0e352ac518fe56a74e34d87051c4e31749/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/7f3c0ede9b031d4a82d61800903dc64950c8d68f6c29593995241de7dd0f5bb4/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/bbedf6923a8ab23a1ff54e4ff1e0965f5dfa5e7e95d8f1372b619a94be39f9da/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/b72904b0d89b847e11bd298e605dc4488230b873c7006a3ff71ce7beb1473e25/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/7d7fbbff0b9f102022f797c4e6c83430389b43348fa6a68e4fc3ba6b1d09f8e1/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/f3e03922029e56d65c90124ddb888a7e1653dd4f8e86863806a6f6d8e56f2ca6/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/fe5e9d7cc562b294708486ffaafddbf717c1fce444176047167691254aab3d8e/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/56bfa59d05c7058704658413b486b6dd56e3609b03b1319f5157d6d1f299d483/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/0117b6388db500f453f5321d423072c4d5e085bcde2153ca556399ea1c1f09d4/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/7f93ab76e53199b7b9cee84b05968720bac59a916c7e48e060f114667e670f17/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/f8c28ac30e567b79a7c44ce7d0770535ebf2505d481d109c4e4e9a31e236aea2/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/e19014fdada7322e738fd542047a4e35c84e97af41c4cf6adb52ca2d95c497a7/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/d94941c99accef0ba5833a0bad903b9c8cff30678bd4f61bac933f1bb2b6026c/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/5d5bb75e824d418ea2b1f3622c188ca0e82437764c63dca5a3a512982766f974/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/098055e7019734bc564747d15c2e5b5f1b78a9e3d3f6c303c48c9dbc010fb370/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/8e5a341178df5f3cc4f8dc0b380d5744a5407d447cb2c1d57d2c884fb930102d/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/b4fc820782b887329c25d480a44b16229736302152187a9440b88525a2354689/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/1d9a1eabd5ad38c51d15992e9e24ad9e619361ca4f30e0b4246a7abcd9e97261/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/21a8ff8d9de37e12fbc043ef899e25511413fbfa68d5f5f2e5ac1f6fc947bf77/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7c1d8521b595fcb1b7f0eb10e2c4b01403b7e9c73a911414254abb1231398552/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5b405ed8eebc6eb96e56fead93c89eb8d9b2d032451a15307e3d15c4e82dd789/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3ce8171078855369cdddc713323149577524b85f5df98821969a6b1198c3677e/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/43c38fd32d4b85028c1e2da4c27f054bcdee4467a1a79e71bead14345960448c/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/e57350a2972a584fdfcbd635bfacd13bfbd2af05a77cf4f7e2bc95834d0d07e1/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/1a29f4448e39a1c05f81539f432a1170ea09c6612f6d6c4c60bf63b0267f50ba/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ab1ae997c5b26da47fb5506408cb7be29ac25cf9af7bfbd2b38857a55875397f/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ded27fed5a99d6e47176c326e1822cdcca5fc3fcb607271c99e5c90858186a62/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/5d47e8391665c9a398416cffb1378f57146ce9797c43e7575b2d6df4344a6c8a/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 17:52:21.122511 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.122146 4883 manager.go:217] Machine: {Timestamp:2026-10-30 17:52:21.121644889 +0000 UTC m=+0.078648716 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3138863 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2640a9e10de29d989d3321f2edf96b SystemUUID:ec2640a9-e10d-e29d-989d-3321f2edf96b BootID:80b88d9f-8980-4d85-a6e8-b7efa1c25c39 Filesystems:[{Device:/var/lib/kubelet/pods/4260cb2f-efee-4b45-9f73-a48757f3ce75/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed98329f6d9d0ac3f0bb6062d734f0d64e2e785e26f672905bb7532f1de9b0f8/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c7d6cbe457d11143c85e47449fd7800aceaf613a7a8012fe59ca7b1299a4848/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4c590f5-af8b-4994-8b03-7f5cee045767/volumes/kubernetes.io~projected/kube-api-access-692s4 DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ba65a282-0992-49ce-b4bc-108c8f69016d/volumes/kubernetes.io~projected/kube-api-access-2dxbs DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5/volumes/kubernetes.io~projected/kube-api-access-pmtbq DeviceMajor:0 DeviceMinor:213 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3bf4f7b93de577cc28bf4cdd4b632aaf0262666f691543b4e172102314928297/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0825799559f49d5f45e640d862efec0735c7504f44a3954d773d8e6f8225d9e3/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e94fb26-859a-48d2-a493-200543aa1881/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/07abda71-7d42-43e6-9878-90ab7da7d9c4/volumes/kubernetes.io~projected/kube-api-access-xlkgn DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a9bbea95be20bac1e1bb2c2164a2994870d7dee69975397b9cdf1131560a066/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a672c72572bbbc22e25fbf1853e3e557e72d76e81375d5db4e9e71412af9561/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4260cb2f-efee-4b45-9f73-a48757f3ce75/volumes/kubernetes.io~projected/kube-api-access-zr6bn DeviceMajor:0 DeviceMinor:210 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3d610c80-f954-4f69-ab9a-e75c56c98652/volumes/kubernetes.io~projected/kube-api-access-gv255 DeviceMajor:0 DeviceMinor:274 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afde3c89beda09c91adcc60ebd6d6823c774cdc8ae9d4a0911a77f5f21d8c65a/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6956a31-29a1-4cd7-bc8b-9cda43dc2e73/volumes/kubernetes.io~projected/kube-api-access-4xx66 DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1aa6b58-d1c3-40a0-873c-86b44c438fac/volumes/kubernetes.io~projected/kube-api-access-j9pq9 DeviceMajor:0 DeviceMinor:215 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfd8e5655731d2860f1ddab8ce6a6ddb225411b8348500bbf21f6e92f55f1903/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/19bd958f-7d20-4da4-818f-06e23e01ab37/volumes/kubernetes.io~projected/kube-api-access-rtx8j DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1aa6b58-d1c3-40a0-873c-86b44c438fac/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0cd0faf02ac1d0545ae1275e67b860caa84ccb64b1c0455eab979dbaad30084b/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/15f0fc8a-91c4-404a-91ec-b39914e83cae/volumes/kubernetes.io~projected/kube-api-access-w9756 DeviceMajor:0 DeviceMinor:211 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4aa5e79-72a0-4b20-a639-ac9b61be4759/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d11ff40ffeed3a7b885575145b1a5889d89641ddf5c5dc4bde2ebf43e652912/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5d1adae-1728-499a-9d16-ec24f1d69160/volumes/kubernetes.io~projected/kube-api-access-4s2j6 DeviceMajor:0 DeviceMinor:270 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:269 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4c590f5-af8b-4994-8b03-7f5cee045767/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3952840a381d0ce2f4744ec7477a9cf0c6c5fae507720774505f073f2d2e8bbb/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f5d511d548c2fdefd45d8bdd0b8065dfcfacab0f31ac1c49fd7a36f41b150e3/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7cf0aaaf-7ca8-44b7-afad-829e4455e56a/volumes/kubernetes.io~projected/kube-api-access-88w5h DeviceMajor:0 DeviceMinor:214 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2cdaadf0279074953b6cff64d8032e4817e8b7d1b9e297ce3e09dbab274dd9a0/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3f27e6dc43e7301384b86d9bda5ce54b9cea35fe48942ba6912fffa683f275b/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19cd058eb83329fa6bc4277bf0e5e8faa72e400ef58a8faa61470589005876c9/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d4c590f5-af8b-4994-8b03-7f5cee045767/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea379eb-5288-4f0f-90f4-712600a2fc01/volumes/kubernetes.io~projected/kube-api-access-x8v7l DeviceMajor:0 DeviceMinor:296 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/06498a3b-c75b-4bf5-9fc7-beced86dbfc5/volumes/kubernetes.io~projected/kube-api-access-gtlqq DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5e94fb26-859a-48d2-a493-200543aa1881/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9d6f83f6-600c-4fa7-a20b-6a179c69bffc/volumes/kubernetes.io~projected/kube-api-access-j6mwh DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/97c8fb7c-e9a6-4eff-9a66-b70848c03a95/volumes/kubernetes.io~projected/kube-api-access-plhvw DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0/volumes/kubernetes.io~projected/kube-api-access-mgj9b DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4aa5e79-72a0-4b20-a639-ac9b61be4759/volumes/kubernetes.io~projected/kube-api-access-nf98n DeviceMajor:0 DeviceMinor:209 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8f683348012bd6873a25ecaf7c515e4200ff75713d9f8105822f8efbec1d83f/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa/volumes/kubernetes.io~projected/kube-api-access-sr2r4 DeviceMajor:0 DeviceMinor:295 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881/volumes/kubernetes.io~projected/kube-api-access-r6kzk DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3866e6ffabee47730af52b48828cedea8224c425c2f6bcbf19d40c024946200e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15f0fc8a-91c4-404a-91ec-b39914e83cae/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4fb42c05-fb07-4dae-975b-71d45dd52e03/volumes/kubernetes.io~projected/kube-api-access-67r2z DeviceMajor:0 DeviceMinor:212 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f8dee39d1523e2e2efed08471bf46398d73d72028d5b0edae754cac4eb95746/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4fb42c05-fb07-4dae-975b-71d45dd52e03/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/867cb674-bd6b-4ca7-98a6-fa31bdaa132d/volumes/kubernetes.io~projected/kube-api-access-bwdl6 DeviceMajor:0 DeviceMinor:208 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e45d20e6-c15c-4f25-9c13-d9cebc4775f0/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:303 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9f90e7f00f9665d52b7a1c5f5b916dbc31411be8d913d761cf750fc3e6f3ede/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4260cb2f-efee-4b45-9f73-a48757f3ce75/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff8e2d857292e8512cc53593b73ef901f6c336dc2f9a9b24e074830970341052/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e45d20e6-c15c-4f25-9c13-d9cebc4775f0/volumes/kubernetes.io~projected/kube-api-access-nslxb DeviceMajor:0 DeviceMinor:308 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2ce04018-ba31-464c-8996-967b9100f174/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5e94fb26-859a-48d2-a493-200543aa1881/volumes/kubernetes.io~projected/kube-api-access-vmnr8 DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/617a558f570813a7e1e53a91fc54d214c8cec4ac686dd1a8a6459822de259180/userdata/shm DeviceMajor:0 DeviceMinor:309 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06ec5fb6cb500a01567f6fcd99b57082fda5e62cb8a7adadbb3af9853bd1f97f/userdata/shm DeviceMajor:0 DeviceMinor:283 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d1e25bb5-1c5b-449e-a7f0-0deb010035ed/volumes/kubernetes.io~projected/kube-api-access-h54zm DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:06ec5fb6cb500a0 MacAddress:da:d1:66:0e:ff:1f Speed:10000 Mtu:8901} {Name:0c7d6cbe457d111 MacAddress:6e:65:d6:e2:96:73 Speed:10000 Mtu:8901} {Name:0d11ff40ffeed3a MacAddress:aa:55:d8:2e:18:52 Speed:10000 Mtu:8901} {Name:1a672c72572bbbc MacAddress:f2:b4:b4:cc:e2:ca Speed:10000 Mtu:8901} {Name:2cdaadf02790749 MacAddress:e2:00:00:93:22:76 Speed:10000 Mtu:8901} {Name:2f5d511d548c2fd MacAddress:82:f3:a8:8d:18:a6 Speed:10000 Mtu:8901} {Name:3a9bbea95be20ba MacAddress:8a:43:74:bb:40:79 Speed:10000 Mtu:8901} {Name:3bf4f7b93de577c MacAddress:7e:79:01:72:45:9d Speed:10000 Mtu:8901} {Name:617a558f570813a MacAddress:a6:f4:5c:5c:e0:38 Speed:10000 Mtu:8901} {Name:afde3c89beda09c MacAddress:26:76:82:04:ac:bc Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d3:66:b1:d3 Speed:0 Mtu:9001} {Name:br-int MacAddress:da:ef:5f:df:a9:4f Speed:0 Mtu:8901} {Name:ed98329f6d9d0ac MacAddress:1e:ad:73:4d:71:5e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d3:66:b1:d3 Speed:0 Mtu:9001} {Name:ff8e2d857292e85 MacAddress:de:d2:d3:63:58:de Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:6a:23:22:19:7f:6b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:62:5b:5e:33:2c:05 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:52:21.122511 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.122513 4883 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:52:21.122783 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.122561 4883 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:52:21.122943 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.122907 4883 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:52:21.123151 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.122957 4883 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-250.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:52:21.123197 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.123156 4883 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:52:21.123197 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.123165 4883 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:52:21.123197 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.123177 4883 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:52:21.123277 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.123200 4883 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:52:21.123277 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.123225 4883 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:52:21.123442 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.123432 4883 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:52:21.123563 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.123554 4883 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:52:21.123599 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.123591 4883 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:52:21.123630 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.123607 4883 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:52:21.123630 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.123620 4883 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:52:21.123692 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.123633 4883 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:52:21.123692 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.123642 4883 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:52:21.124625 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.124609 4883 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:52:21.124625 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.124623 4883 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:52:21.125247 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.125225 4883 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:52:21.127537 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.127520 4883 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:52:21.127537 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.127534 4883 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:52:21.127715 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.127702 4883 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 17:52:21.127957 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.127927 4883 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:52:21.128032 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.127973 4883 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:52:21.128100 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128090 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128108 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128118 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128124 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128130 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128135 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128141 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128147 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128154 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128160 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128169 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128178 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128229 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:52:21.128246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.128235 4883 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:52:21.131616 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.131605 4883 server.go:1263] "Started kubelet" Oct 30 17:52:21.131685 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.131655 4883 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:52:21.132249 ip-10-0-130-250 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:52:21.132927 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.132773 4883 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:52:21.132927 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.132869 4883 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:52:21.133411 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.133388 4883 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:52:21.133411 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.133406 4883 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.137419 4883 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.137425 4883 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.137511 4883 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:46:42 +0000 UTC" deadline="2028-05-28 16:58:11.696389898 +0000 UTC" Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.137554 4883 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13823h5m50.558841488s" Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.137587 4883 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.137595 4883 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:21.137682 4883 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-250.ec2.internal\" not found" Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.137751 4883 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.138757 4883 factory.go:55] Registering systemd factory Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.138774 4883 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.139195 4883 factory.go:153] Registering CRI-O factory Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.139215 4883 factory.go:223] Registration of the crio container factory successfully Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.139291 4883 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.139313 4883 factory.go:103] Registering Raw factory Oct 30 17:52:21.139463 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.139333 4883 manager.go:1199] Started watching for new ooms in manager Oct 30 17:52:21.140271 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.139766 4883 manager.go:319] Starting recovery of all containers Oct 30 17:52:21.142399 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142348 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4aa5e79-72a0-4b20-a639-ac9b61be4759" volumeName="kubernetes.io/configmap/e4aa5e79-72a0-4b20-a639-ac9b61be4759-config" seLinuxMountContext="" Oct 30 17:52:21.142526 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142401 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44ce8b1c-6ab2-4f83-a1f6-183cc6cea104" volumeName="kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret" seLinuxMountContext="" Oct 30 17:52:21.142526 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142418 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4c590f5-af8b-4994-8b03-7f5cee045767" volumeName="kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-ovnkube-config" seLinuxMountContext="" Oct 30 17:52:21.142526 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142434 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4c590f5-af8b-4994-8b03-7f5cee045767" volumeName="kubernetes.io/secret/d4c590f5-af8b-4994-8b03-7f5cee045767-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 17:52:21.142526 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142448 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4aa5e79-72a0-4b20-a639-ac9b61be4759" volumeName="kubernetes.io/projected/e4aa5e79-72a0-4b20-a639-ac9b61be4759-kube-api-access-nf98n" seLinuxMountContext="" Oct 30 17:52:21.142526 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142461 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07abda71-7d42-43e6-9878-90ab7da7d9c4" volumeName="kubernetes.io/configmap/07abda71-7d42-43e6-9878-90ab7da7d9c4-cni-binary-copy" seLinuxMountContext="" Oct 30 17:52:21.142526 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142475 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15f0fc8a-91c4-404a-91ec-b39914e83cae" volumeName="kubernetes.io/empty-dir/15f0fc8a-91c4-404a-91ec-b39914e83cae-tmp" seLinuxMountContext="" Oct 30 17:52:21.142526 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142487 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ce04018-ba31-464c-8996-967b9100f174" volumeName="kubernetes.io/configmap/2ce04018-ba31-464c-8996-967b9100f174-konnectivity-ca" seLinuxMountContext="" Oct 30 17:52:21.142526 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142503 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea379eb-5288-4f0f-90f4-712600a2fc01" volumeName="kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-trusted-ca-bundle" seLinuxMountContext="" Oct 30 17:52:21.142526 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142518 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e94fb26-859a-48d2-a493-200543aa1881" volumeName="kubernetes.io/projected/5e94fb26-859a-48d2-a493-200543aa1881-kube-api-access-vmnr8" seLinuxMountContext="" Oct 30 17:52:21.142526 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142529 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa" volumeName="kubernetes.io/projected/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-api-access-sr2r4" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142541 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba65a282-0992-49ce-b4bc-108c8f69016d" volumeName="kubernetes.io/projected/ba65a282-0992-49ce-b4bc-108c8f69016d-kube-api-access-2dxbs" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142553 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e45d20e6-c15c-4f25-9c13-d9cebc4775f0" volumeName="kubernetes.io/configmap/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-signing-cabundle" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142564 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07abda71-7d42-43e6-9878-90ab7da7d9c4" volumeName="kubernetes.io/configmap/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-daemon-config" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142580 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07abda71-7d42-43e6-9878-90ab7da7d9c4" volumeName="kubernetes.io/projected/07abda71-7d42-43e6-9878-90ab7da7d9c4-kube-api-access-xlkgn" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142591 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cf0aaaf-7ca8-44b7-afad-829e4455e56a" volumeName="kubernetes.io/configmap/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-config-volume" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142603 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="867cb674-bd6b-4ca7-98a6-fa31bdaa132d" volumeName="kubernetes.io/projected/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-kube-api-access-bwdl6" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142614 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d6f83f6-600c-4fa7-a20b-6a179c69bffc" volumeName="kubernetes.io/projected/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-kube-api-access-j6mwh" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142626 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d610c80-f954-4f69-ab9a-e75c56c98652" volumeName="kubernetes.io/projected/3d610c80-f954-4f69-ab9a-e75c56c98652-kube-api-access-gv255" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142637 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa" volumeName="kubernetes.io/empty-dir/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-data-volume" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142650 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6956a31-29a1-4cd7-bc8b-9cda43dc2e73" volumeName="kubernetes.io/projected/b6956a31-29a1-4cd7-bc8b-9cda43dc2e73-kube-api-access-4xx66" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142663 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f8536c2-cd14-4d88-ab1d-63a2f032aeb5" volumeName="kubernetes.io/configmap/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-config" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142689 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea379eb-5288-4f0f-90f4-712600a2fc01" volumeName="kubernetes.io/empty-dir/4ea379eb-5288-4f0f-90f4-712600a2fc01-archives-path" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142700 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1e25bb5-1c5b-449e-a7f0-0deb010035ed" volumeName="kubernetes.io/configmap/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-iptables-alerter-script" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142734 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4260cb2f-efee-4b45-9f73-a48757f3ce75" volumeName="kubernetes.io/projected/4260cb2f-efee-4b45-9f73-a48757f3ce75-kube-api-access-zr6bn" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142747 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1aa6b58-d1c3-40a0-873c-86b44c438fac" volumeName="kubernetes.io/secret/c1aa6b58-d1c3-40a0-873c-86b44c438fac-cert" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142760 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e94fb26-859a-48d2-a493-200543aa1881" volumeName="kubernetes.io/empty-dir/5e94fb26-859a-48d2-a493-200543aa1881-etc-tuned" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142772 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" volumeName="kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142790 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e45d20e6-c15c-4f25-9c13-d9cebc4775f0" volumeName="kubernetes.io/secret/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-signing-key" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142802 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15f0fc8a-91c4-404a-91ec-b39914e83cae" volumeName="kubernetes.io/configmap/15f0fc8a-91c4-404a-91ec-b39914e83cae-service-ca-bundle" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142831 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ce04018-ba31-464c-8996-967b9100f174" volumeName="kubernetes.io/secret/2ce04018-ba31-464c-8996-967b9100f174-agent-certs" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142843 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fb42c05-fb07-4dae-975b-71d45dd52e03" volumeName="kubernetes.io/projected/4fb42c05-fb07-4dae-975b-71d45dd52e03-kube-api-access-67r2z" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142860 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e45d20e6-c15c-4f25-9c13-d9cebc4775f0" volumeName="kubernetes.io/projected/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-kube-api-access-nslxb" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142872 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15f0fc8a-91c4-404a-91ec-b39914e83cae" volumeName="kubernetes.io/secret/15f0fc8a-91c4-404a-91ec-b39914e83cae-serving-cert" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142886 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f8536c2-cd14-4d88-ab1d-63a2f032aeb5" volumeName="kubernetes.io/projected/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-kube-api-access-pmtbq" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142898 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4260cb2f-efee-4b45-9f73-a48757f3ce75" volumeName="kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-default-certificate" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142912 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea379eb-5288-4f0f-90f4-712600a2fc01" volumeName="kubernetes.io/projected/4ea379eb-5288-4f0f-90f4-712600a2fc01-kube-api-access-x8v7l" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142924 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="867cb674-bd6b-4ca7-98a6-fa31bdaa132d" volumeName="kubernetes.io/configmap/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-telemetry-config" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142977 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1aa6b58-d1c3-40a0-873c-86b44c438fac" volumeName="kubernetes.io/projected/c1aa6b58-d1c3-40a0-873c-86b44c438fac-kube-api-access-j9pq9" seLinuxMountContext="" Oct 30 17:52:21.143013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.142990 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4aa5e79-72a0-4b20-a639-ac9b61be4759" volumeName="kubernetes.io/secret/e4aa5e79-72a0-4b20-a639-ac9b61be4759-serving-cert" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143037 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15f0fc8a-91c4-404a-91ec-b39914e83cae" volumeName="kubernetes.io/configmap/15f0fc8a-91c4-404a-91ec-b39914e83cae-trusted-ca-bundle" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143049 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fb42c05-fb07-4dae-975b-71d45dd52e03" volumeName="kubernetes.io/configmap/4fb42c05-fb07-4dae-975b-71d45dd52e03-config" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143061 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cf0aaaf-7ca8-44b7-afad-829e4455e56a" volumeName="kubernetes.io/empty-dir/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-tmp-dir" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143073 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cf0aaaf-7ca8-44b7-afad-829e4455e56a" volumeName="kubernetes.io/projected/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-kube-api-access-88w5h" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143085 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1e25bb5-1c5b-449e-a7f0-0deb010035ed" volumeName="kubernetes.io/projected/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-kube-api-access-h54zm" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143097 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19bd958f-7d20-4da4-818f-06e23e01ab37" volumeName="kubernetes.io/projected/19bd958f-7d20-4da4-818f-06e23e01ab37-kube-api-access-rtx8j" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143107 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4260cb2f-efee-4b45-9f73-a48757f3ce75" volumeName="kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-stats-auth" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143118 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fb42c05-fb07-4dae-975b-71d45dd52e03" volumeName="kubernetes.io/secret/4fb42c05-fb07-4dae-975b-71d45dd52e03-serving-cert" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143131 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e94fb26-859a-48d2-a493-200543aa1881" volumeName="kubernetes.io/empty-dir/5e94fb26-859a-48d2-a493-200543aa1881-tmp" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143144 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" volumeName="kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cni-binary-copy" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143158 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" volumeName="kubernetes.io/projected/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-kube-api-access-r6kzk" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143172 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0" volumeName="kubernetes.io/projected/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-kube-api-access-mgj9b" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143184 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4c590f5-af8b-4994-8b03-7f5cee045767" volumeName="kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-env-overrides" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143196 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06498a3b-c75b-4bf5-9fc7-beced86dbfc5" volumeName="kubernetes.io/projected/06498a3b-c75b-4bf5-9fc7-beced86dbfc5-kube-api-access-gtlqq" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143210 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15f0fc8a-91c4-404a-91ec-b39914e83cae" volumeName="kubernetes.io/empty-dir/15f0fc8a-91c4-404a-91ec-b39914e83cae-snapshots" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143222 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" volumeName="kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143243 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97c8fb7c-e9a6-4eff-9a66-b70848c03a95" volumeName="kubernetes.io/projected/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-kube-api-access-plhvw" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143254 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4c590f5-af8b-4994-8b03-7f5cee045767" volumeName="kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-ovnkube-script-lib" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143265 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4c590f5-af8b-4994-8b03-7f5cee045767" volumeName="kubernetes.io/projected/d4c590f5-af8b-4994-8b03-7f5cee045767-kube-api-access-692s4" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143279 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15f0fc8a-91c4-404a-91ec-b39914e83cae" volumeName="kubernetes.io/projected/15f0fc8a-91c4-404a-91ec-b39914e83cae-kube-api-access-w9756" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143293 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea379eb-5288-4f0f-90f4-712600a2fc01" volumeName="kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-service-ca-bundle" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143306 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0" volumeName="kubernetes.io/configmap/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-serviceca" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143320 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fb42c05-fb07-4dae-975b-71d45dd52e03" volumeName="kubernetes.io/configmap/4fb42c05-fb07-4dae-975b-71d45dd52e03-trusted-ca" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143333 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97c8fb7c-e9a6-4eff-9a66-b70848c03a95" volumeName="kubernetes.io/empty-dir/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-tmp-dir" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143345 4883 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d1adae-1728-499a-9d16-ec24f1d69160" volumeName="kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6" seLinuxMountContext="" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143356 4883 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:52:21.147219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.143362 4883 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:52:21.160081 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.160008 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/1.log" Oct 30 17:52:21.161469 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.161006 4883 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:52:21.163709 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.163385 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/1.log" Oct 30 17:52:21.170037 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.170010 4883 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:52:21.170037 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.170035 4883 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:52:21.170202 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.170057 4883 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:52:21.170202 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:21.170100 4883 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 17:52:21.173118 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.173095 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 17:52:21.175345 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.175269 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 17:52:21.194748 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.194698 4883 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 17:52:21.194999 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.194975 4883 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:52:21.208029 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.207990 4883 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:52:21.208184 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.208040 4883 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:52:21.208453 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.208434 4883 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:52:21.213115 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.213082 4883 generic.go:356] "Generic (PLEG): container finished" podID="49b07006ee51f49de29dbabbdd01d907" containerID="a16a80d5d5df75babf88618464c6c255ad9c9ea0851d381dde490fb683381a6a" exitCode=0 Oct 30 17:52:21.216250 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.216227 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/1.log" Oct 30 17:52:21.216346 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.216260 4883 generic.go:356] "Generic (PLEG): container finished" podID="4fb42c05-fb07-4dae-975b-71d45dd52e03" containerID="9d4cc54e9eac0995064b052a1728805f57581f4f07a54b60d85f9eddfac462f5" exitCode=255 Oct 30 17:52:21.222499 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.222472 4883 generic.go:356] "Generic (PLEG): container finished" podID="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" containerID="9e3e52a68b94c103eaded9942a7bd9f499f985c596d74c9b37a974444566a05e" exitCode=0 Oct 30 17:52:21.222499 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.222494 4883 generic.go:356] "Generic (PLEG): container finished" podID="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" containerID="55e18ae0b3eb8a4053a504a1f2e7a67f7a444cf735fc8d0437529b2cde94cb5b" exitCode=0 Oct 30 17:52:21.222499 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.222500 4883 generic.go:356] "Generic (PLEG): container finished" podID="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" containerID="af69146a931f76689cafde2179fe4cc5a32bb150be5580390cd4484689ff8588" exitCode=0 Oct 30 17:52:21.222499 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.222506 4883 generic.go:356] "Generic (PLEG): container finished" podID="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" containerID="20692f6b55bd7ee409ad0dca5174951273e19b47d9350d55046a0379d1e2714b" exitCode=0 Oct 30 17:52:21.222782 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.222514 4883 generic.go:356] "Generic (PLEG): container finished" podID="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" containerID="4206893f44bad5f9224c0a8780dabcd4d9f7a884845cf858c6a9adcd4d03926e" exitCode=0 Oct 30 17:52:21.222782 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.222520 4883 generic.go:356] "Generic (PLEG): container finished" podID="7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881" containerID="522af28188fcd61bec9bf1f3f5ee9a993d21446672c906e96deb48843b1fd4af" exitCode=0 Oct 30 17:52:21.228238 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.228218 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 17:52:21.228599 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.228580 4883 generic.go:356] "Generic (PLEG): container finished" podID="d4c590f5-af8b-4994-8b03-7f5cee045767" containerID="e6339ce54b85a59f0f0bf4ae406ad5739c82d459b07af86b0ccf0dcc93b9cb10" exitCode=1 Oct 30 17:52:21.241442 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.241382 4883 manager.go:324] Recovery completed Oct 30 17:52:21.261196 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.261165 4883 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:52:21.261196 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.261187 4883 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:52:21.261370 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.261210 4883 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:52:21.261370 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.261338 4883 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 17:52:21.261824 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.261349 4883 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 17:52:21.261864 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.261828 4883 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 17:52:21.261864 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.261835 4883 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 17:52:21.261864 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.261841 4883 policy_none.go:50] "Start" Oct 30 17:52:21.261864 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.261851 4883 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:52:21.261864 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.261864 4883 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:52:21.262029 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.262019 4883 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 17:52:21.262059 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.262030 4883 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 17:52:21.262059 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.262036 4883 policy_none.go:44] "Start" Oct 30 17:52:21.266381 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.266348 4883 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:52:21.269825 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.269807 4883 manager.go:341] "Starting Device Plugin manager" Oct 30 17:52:21.269909 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:21.269835 4883 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:52:21.269909 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.269843 4883 server.go:87] "Starting device plugin registration server" Oct 30 17:52:21.270179 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.270166 4883 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:52:21.270217 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.270181 4883 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:52:21.270282 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.270265 4883 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:52:21.270408 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.270363 4883 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal"] Oct 30 17:52:21.270491 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.270366 4883 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:52:21.270537 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.270504 4883 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:52:21.270537 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.270513 4883 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:52:21.270629 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.270547 4883 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:52:21.270488542Z","UUID":"bc715b74-1b83-4b67-8733-ee84c6e115ba","Handler":null,"Name":"","Endpoint":""} Oct 30 17:52:21.271088 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.271043 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" event={"ID":"ef96f59234cd19c8b7af6a47596c7964","Type":"ContainerStarted","Data":"26edd1c4daca79cc59d6fea149e66df7a414a32235206f891be900636bc3bc1d"} Oct 30 17:52:21.271201 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.271185 4883 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.271265 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.271224 4883 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.271347 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.271188 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" event={"ID":"ef96f59234cd19c8b7af6a47596c7964","Type":"ContainerStarted","Data":"cfd8e5655731d2860f1ddab8ce6a6ddb225411b8348500bbf21f6e92f55f1903"} Oct 30 17:52:21.271441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.271412 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" event={"ID":"49b07006ee51f49de29dbabbdd01d907","Type":"ContainerStarted","Data":"e5f4edf4cfa93f2395a11111ca9f38236dc5a0db0248a3a2fdbdf420a7bf490b"} Oct 30 17:52:21.271513 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.271440 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" event={"ID":"49b07006ee51f49de29dbabbdd01d907","Type":"ContainerDied","Data":"a16a80d5d5df75babf88618464c6c255ad9c9ea0851d381dde490fb683381a6a"} Oct 30 17:52:21.271513 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.271461 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" event={"ID":"49b07006ee51f49de29dbabbdd01d907","Type":"ContainerStarted","Data":"c3f27e6dc43e7301384b86d9bda5ce54b9cea35fe48942ba6912fffa683f275b"} Oct 30 17:52:21.273026 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:21.272621 4883 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:52:21.273834 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.273773 4883 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:52:21.273834 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.273827 4883 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:52:21.294274 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.294246 4883 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:52:21.294441 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:21.294308 4883 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.294441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.294254 4883 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:52:21.294441 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:21.294378 4883 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-250.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.371269 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.371233 4883 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:52:21.372867 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.372846 4883 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:52:21.373006 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.372899 4883 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:52:21.373006 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.372913 4883 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-250.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:52:21.373006 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.372986 4883 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.395065 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.395035 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/49b07006ee51f49de29dbabbdd01d907-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" (UID: \"49b07006ee51f49de29dbabbdd01d907\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.395191 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.395116 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49b07006ee51f49de29dbabbdd01d907-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" (UID: \"49b07006ee51f49de29dbabbdd01d907\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.395191 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.395162 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef96f59234cd19c8b7af6a47596c7964-config\") pod \"kube-apiserver-proxy-ip-10-0-130-250.ec2.internal\" (UID: \"ef96f59234cd19c8b7af6a47596c7964\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.409064 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.409038 4883 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.409174 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.409121 4883 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.495355 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.495273 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef96f59234cd19c8b7af6a47596c7964-config\") pod \"kube-apiserver-proxy-ip-10-0-130-250.ec2.internal\" (UID: \"ef96f59234cd19c8b7af6a47596c7964\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.495355 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.495307 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/49b07006ee51f49de29dbabbdd01d907-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" (UID: \"49b07006ee51f49de29dbabbdd01d907\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.495355 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.495327 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49b07006ee51f49de29dbabbdd01d907-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" (UID: \"49b07006ee51f49de29dbabbdd01d907\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.495574 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.495378 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/49b07006ee51f49de29dbabbdd01d907-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" (UID: \"49b07006ee51f49de29dbabbdd01d907\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.495574 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.495404 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49b07006ee51f49de29dbabbdd01d907-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" (UID: \"49b07006ee51f49de29dbabbdd01d907\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:52:21.495574 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:21.495378 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef96f59234cd19c8b7af6a47596c7964-config\") pod \"kube-apiserver-proxy-ip-10-0-130-250.ec2.internal\" (UID: \"ef96f59234cd19c8b7af6a47596c7964\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:52:22.124086 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.124051 4883 apiserver.go:52] "Watching apiserver" Oct 30 17:52:22.187347 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.187319 4883 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:52:22.188118 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.188092 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-xbxvk","openshift-image-registry/node-ca-7gt8q","openshift-insights/insights-runtime-extractor-sw2jb","openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm","openshift-cluster-node-tuning-operator/tuned-dpch6","openshift-ingress/router-default-f6d947f75-ln5z8","openshift-multus/network-metrics-daemon-dl2lm","openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh","kube-system/kas-connection-checker-685895c7c9-wjjnl","openshift-ovn-kubernetes/ovnkube-node-k6zpc","openshift-dns/dns-default-6fncr","openshift-ingress-canary/ingress-canary-t5n25","openshift-multus/multus-md4zx","kube-system/global-pull-secret-syncer-fglxs","openshift-console-operator/console-operator-7cd56b8dbc-5zcf9","openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26","kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal","openshift-insights/insights-operator-7c4986d449-csr4p","kube-system/konnectivity-agent-vdlbp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m","kube-system/kas-connection-checker-685895c7c9-2tjnd","openshift-network-diagnostics/network-check-target-vqltd","openshift-multus/multus-additional-cni-plugins-2sfml","openshift-insights/periodic-gathering-x2wf2-g8lv5","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt","openshift-network-operator/iptables-alerter-wbzt5","openshift-service-ca/service-ca-5fffc97bdf-wql78","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal"] Oct 30 17:52:22.191906 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.191611 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:22.192673 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.192646 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:22.193348 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.193328 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.197243 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.197225 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:22.199110 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.199093 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:22.200564 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.200545 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.204868 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.204849 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.204868 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.204853 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.206390 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.206378 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.206482 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.206471 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.206552 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.206536 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.206582 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.206541 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 17:52:22.206582 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.206576 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 17:52:22.206908 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.206897 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.208129 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.208108 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.208210 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.208155 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 17:52:22.208210 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.208170 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 17:52:22.208381 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.208365 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 17:52:22.221362 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.221336 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.221574 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.221558 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.221637 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.221558 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 17:52:22.221637 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.221594 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 17:52:22.221826 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.221807 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 17:52:22.221877 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.221859 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 17:52:22.221945 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.221902 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 17:52:22.222069 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.222054 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.222330 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.222310 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.222509 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.222493 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 17:52:22.236308 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.236286 4883 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:52:22.236413 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.236400 4883 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:52:22.239914 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.239898 4883 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:52:22.257720 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.257703 4883 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:52:22.257794 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.257742 4883 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-250.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-250.ec2.internal" Oct 30 17:52:22.258437 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.258423 4883 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:52:22.258473 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.258453 4883 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-250.ec2.internal" Oct 30 17:52:22.299159 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299134 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.299278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299167 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlkgn\" (UniqueName: \"kubernetes.io/projected/07abda71-7d42-43e6-9878-90ab7da7d9c4-kube-api-access-xlkgn\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.299278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299186 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2j6\" (UniqueName: \"kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6\") pod \"network-check-target-vqltd\" (UID: \"d5d1adae-1728-499a-9d16-ec24f1d69160\") " pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:22.299278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299203 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:22.299278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299257 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtbq\" (UniqueName: \"kubernetes.io/projected/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-kube-api-access-pmtbq\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:22.299473 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299320 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-device-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.299473 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299456 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv255\" (UniqueName: \"kubernetes.io/projected/3d610c80-f954-4f69-ab9a-e75c56c98652-kube-api-access-gv255\") pod \"migrator-66df5d496b-n52mh\" (UID: \"3d610c80-f954-4f69-ab9a-e75c56c98652\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh" Oct 30 17:52:22.299579 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299560 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysconfig\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.299631 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299589 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ce04018-ba31-464c-8996-967b9100f174-konnectivity-ca\") pod \"konnectivity-agent-vdlbp\" (UID: \"2ce04018-ba31-464c-8996-967b9100f174\") " pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:52:22.299631 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299612 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:22.299724 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299658 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fb42c05-fb07-4dae-975b-71d45dd52e03-trusted-ca\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:22.299724 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299684 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.299724 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299702 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/15f0fc8a-91c4-404a-91ec-b39914e83cae-snapshots\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:22.299724 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299719 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmnr8\" (UniqueName: \"kubernetes.io/projected/5e94fb26-859a-48d2-a493-200543aa1881-kube-api-access-vmnr8\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.299912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299734 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtx8j\" (UniqueName: \"kubernetes.io/projected/19bd958f-7d20-4da4-818f-06e23e01ab37-kube-api-access-rtx8j\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:22.299912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299769 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-host\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:52:22.299912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299800 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-system-cni-dir\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.299912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299815 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/15f0fc8a-91c4-404a-91ec-b39914e83cae-snapshots\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:22.299912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299831 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xx66\" (UniqueName: \"kubernetes.io/projected/b6956a31-29a1-4cd7-bc8b-9cda43dc2e73-kube-api-access-4xx66\") pod \"kas-connection-checker-685895c7c9-2tjnd\" (UID: \"b6956a31-29a1-4cd7-bc8b-9cda43dc2e73\") " pod="kube-system/kas-connection-checker-685895c7c9-2tjnd" Oct 30 17:52:22.299912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299859 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-socket-dir-parent\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.300172 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.299958 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-iptables-alerter-script\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:52:22.300172 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300002 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1aa6b58-d1c3-40a0-873c-86b44c438fac-cert\") pod \"ingress-canary-t5n25\" (UID: \"c1aa6b58-d1c3-40a0-873c-86b44c438fac\") " pod="openshift-ingress-canary/ingress-canary-t5n25" Oct 30 17:52:22.300172 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300025 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fb42c05-fb07-4dae-975b-71d45dd52e03-trusted-ca\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:22.300172 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300063 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-netns\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.300285 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300178 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-multus-certs\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.300285 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300232 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-modprobe-d\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.300285 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300259 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15f0fc8a-91c4-404a-91ec-b39914e83cae-serving-cert\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:22.300285 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300275 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692s4\" (UniqueName: \"kubernetes.io/projected/d4c590f5-af8b-4994-8b03-7f5cee045767-kube-api-access-692s4\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.300411 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300301 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:22.300411 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300329 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-daemon-config\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.300411 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300355 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-dbus\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:22.300411 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300377 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysctl-d\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.300411 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300400 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.300585 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300412 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15f0fc8a-91c4-404a-91ec-b39914e83cae-serving-cert\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:22.300585 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300436 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-cni-netd\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.300585 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300479 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6kzk\" (UniqueName: \"kubernetes.io/projected/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-kube-api-access-r6kzk\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.300585 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300534 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:22.300585 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300554 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-crio-socket\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.300585 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300569 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.300820 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300588 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07abda71-7d42-43e6-9878-90ab7da7d9c4-cni-binary-copy\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.300820 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300603 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-kubelet\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.300820 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300640 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4aa5e79-72a0-4b20-a639-ac9b61be4759-config\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:22.300820 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300703 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pq9\" (UniqueName: \"kubernetes.io/projected/c1aa6b58-d1c3-40a0-873c-86b44c438fac-kube-api-access-j9pq9\") pod \"ingress-canary-t5n25\" (UID: \"c1aa6b58-d1c3-40a0-873c-86b44c438fac\") " pod="openshift-ingress-canary/ingress-canary-t5n25" Oct 30 17:52:22.300988 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300862 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtlqq\" (UniqueName: \"kubernetes.io/projected/06498a3b-c75b-4bf5-9fc7-beced86dbfc5-kube-api-access-gtlqq\") pod \"kas-connection-checker-685895c7c9-wjjnl\" (UID: \"06498a3b-c75b-4bf5-9fc7-beced86dbfc5\") " pod="kube-system/kas-connection-checker-685895c7c9-wjjnl" Oct 30 17:52:22.300988 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300869 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e4aa5e79-72a0-4b20-a639-ac9b61be4759-config\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:22.300988 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300899 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-kubelet\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.300988 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.300981 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-node-log\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.301169 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301004 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-signing-cabundle\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:22.301169 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301020 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ce04018-ba31-464c-8996-967b9100f174-agent-certs\") pod \"konnectivity-agent-vdlbp\" (UID: \"2ce04018-ba31-464c-8996-967b9100f174\") " pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:52:22.301169 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301036 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-trusted-ca-bundle\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:22.301169 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301073 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-host\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.301169 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301109 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.301169 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301134 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-env-overrides\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.301418 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301194 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-lib-modules\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.301418 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301223 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.301418 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301271 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-config-volume\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:22.301418 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301300 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhvw\" (UniqueName: \"kubernetes.io/projected/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-kube-api-access-plhvw\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:52:22.301418 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301328 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-trusted-ca-bundle\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:22.301418 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301327 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-etc-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.301418 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301362 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdl6\" (UniqueName: \"kubernetes.io/projected/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-kube-api-access-bwdl6\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:22.301418 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301391 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-cni-bin\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.301418 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301408 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-var-lib-kubelet\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301433 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-slash\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301469 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-run-netns\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301507 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-ovnkube-script-lib\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301531 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-cni-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301546 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-conf-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301562 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4aa5e79-72a0-4b20-a639-ac9b61be4759-serving-cert\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301592 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-log-socket\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301616 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysctl-conf\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301631 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cnibin\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301648 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-systemd-units\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301673 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4c590f5-af8b-4994-8b03-7f5cee045767-ovn-node-metrics-cert\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301695 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301739 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-stats-auth\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301769 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301786 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-cni-bin\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.301805 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301801 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-cnibin\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301820 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301867 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-default-certificate\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301923 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-socket-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301970 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-registration-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301972 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e4aa5e79-72a0-4b20-a639-ac9b61be4759-serving-cert\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.301993 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-tmp-dir\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302045 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302107 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2r4\" (UniqueName: \"kubernetes.io/projected/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-api-access-sr2r4\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302124 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-tmp-dir\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302127 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f0fc8a-91c4-404a-91ec-b39914e83cae-service-ca-bundle\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302186 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-var-lib-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302213 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-sys-fs\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302242 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302245 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f0fc8a-91c4-404a-91ec-b39914e83cae-service-ca-bundle\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302278 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-ovnkube-config\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302315 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e94fb26-859a-48d2-a493-200543aa1881-etc-tuned\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302342 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgj9b\" (UniqueName: \"kubernetes.io/projected/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-kube-api-access-mgj9b\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302358 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4fb42c05-fb07-4dae-975b-71d45dd52e03-serving-cert\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302393 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dxbs\" (UniqueName: \"kubernetes.io/projected/ba65a282-0992-49ce-b4bc-108c8f69016d-kube-api-access-2dxbs\") pod \"volume-data-source-validator-58c9f5f46d-hpv5m\" (UID: \"ba65a282-0992-49ce-b4bc-108c8f69016d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302422 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h54zm\" (UniqueName: \"kubernetes.io/projected/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-kube-api-access-h54zm\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:52:22.302441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302433 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e94fb26-859a-48d2-a493-200543aa1881-etc-tuned\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302438 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-service-ca-bundle\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302509 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-data-volume\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302528 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4fb42c05-fb07-4dae-975b-71d45dd52e03-serving-cert\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302539 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6mwh\" (UniqueName: \"kubernetes.io/projected/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-kube-api-access-j6mwh\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302565 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4fb42c05-fb07-4dae-975b-71d45dd52e03-config\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302592 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ea379eb-5288-4f0f-90f4-712600a2fc01-archives-path\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302598 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-data-volume\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302598 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-service-ca-bundle\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302622 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302655 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88w5h\" (UniqueName: \"kubernetes.io/projected/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-kube-api-access-88w5h\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302675 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ea379eb-5288-4f0f-90f4-712600a2fc01-archives-path\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302754 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4fb42c05-fb07-4dae-975b-71d45dd52e03-config\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302764 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nslxb\" (UniqueName: \"kubernetes.io/projected/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-kube-api-access-nslxb\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302800 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-systemd\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302824 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-cni-multus\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302852 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-kubelet-config\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302854 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302888 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302906 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-hosts-file\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.302969 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f0fc8a-91c4-404a-91ec-b39914e83cae-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303024 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-tmp-dir\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303128 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-k8s-cni-cncf-io\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303135 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-tmp-dir\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303158 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-etc-kubernetes\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.303214 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303182 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-serviceca\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303256 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-host-slash\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303264 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f0fc8a-91c4-404a-91ec-b39914e83cae-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303283 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8v7l\" (UniqueName: \"kubernetes.io/projected/4ea379eb-5288-4f0f-90f4-712600a2fc01-kube-api-access-x8v7l\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303301 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6bn\" (UniqueName: \"kubernetes.io/projected/4260cb2f-efee-4b45-9f73-a48757f3ce75-kube-api-access-zr6bn\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303316 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-signing-key\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303331 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15f0fc8a-91c4-404a-91ec-b39914e83cae-tmp\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303346 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-system-cni-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303370 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303396 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf98n\" (UniqueName: \"kubernetes.io/projected/e4aa5e79-72a0-4b20-a639-ac9b61be4759-kube-api-access-nf98n\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303416 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15f0fc8a-91c4-404a-91ec-b39914e83cae-tmp\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303437 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-ovn\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303533 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303560 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-run\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303574 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-sys\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303635 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e94fb26-859a-48d2-a493-200543aa1881-tmp\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303673 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67r2z\" (UniqueName: \"kubernetes.io/projected/4fb42c05-fb07-4dae-975b-71d45dd52e03-kube-api-access-67r2z\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303690 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e94fb26-859a-48d2-a493-200543aa1881-tmp\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303700 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-hostroot\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303727 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-kubernetes\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303769 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-systemd\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303795 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9756\" (UniqueName: \"kubernetes.io/projected/15f0fc8a-91c4-404a-91ec-b39914e83cae-kube-api-access-w9756\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:22.304035 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.303815 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-run-ovn-kubernetes\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.304715 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.304679 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.305073 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305022 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.305073 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305033 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.305073 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305022 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:52:22.305254 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305095 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:52:22.305254 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305104 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:52:22.305254 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305098 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:52:22.305254 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305095 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.305254 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305200 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.305484 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305302 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.305484 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305323 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:52:22.305484 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305336 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:52:22.305484 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305417 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.305484 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305435 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 17:52:22.305484 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305465 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305474 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305522 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305543 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305585 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305596 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305596 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305599 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305632 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305635 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305669 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305724 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305743 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305750 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305779 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.305865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305786 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.306267 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.305899 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.306267 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.306039 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.306267 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.306061 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 17:52:22.306267 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.306105 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:52:22.306760 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.306748 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.306987 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.306962 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:52:22.306987 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.306975 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:52:22.307123 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.306995 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.307123 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.307037 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:52:22.307123 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.307040 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.307123 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.307043 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.307123 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.307070 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:52:22.309860 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.309843 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:52:22.309961 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.309945 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ce04018-ba31-464c-8996-967b9100f174-konnectivity-ca\") pod \"konnectivity-agent-vdlbp\" (UID: \"2ce04018-ba31-464c-8996-967b9100f174\") " pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:52:22.310343 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310324 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1aa6b58-d1c3-40a0-873c-86b44c438fac-cert\") pod \"ingress-canary-t5n25\" (UID: \"c1aa6b58-d1c3-40a0-873c-86b44c438fac\") " pod="openshift-ingress-canary/ingress-canary-t5n25" Oct 30 17:52:22.310406 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310341 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-iptables-alerter-script\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:52:22.310406 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310362 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:52:22.310479 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310441 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:52:22.310590 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310530 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:52:22.310590 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310570 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:52:22.310703 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310687 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 17:52:22.310753 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310703 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-daemon-config\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.310808 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310647 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:22.310808 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310796 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 17:52:22.310902 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310818 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 17:52:22.310902 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310796 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k56gt\"" Oct 30 17:52:22.310902 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310867 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 17:52:22.311055 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310947 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zlfnf\"" Oct 30 17:52:22.311055 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310948 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:52:22.311055 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310968 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07abda71-7d42-43e6-9878-90ab7da7d9c4-cni-binary-copy\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.311055 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310983 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nb8hn\"" Oct 30 17:52:22.311055 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310992 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-twdt2\"" Oct 30 17:52:22.311055 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311014 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cwftc\"" Oct 30 17:52:22.311327 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311065 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-99rd9\"" Oct 30 17:52:22.311327 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311068 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:52:22.311327 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311085 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-w7rbg\"" Oct 30 17:52:22.311327 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311108 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ddhr5\"" Oct 30 17:52:22.311327 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311126 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jjhvv\"" Oct 30 17:52:22.311327 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311150 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vnlvf\"" Oct 30 17:52:22.311327 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311183 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k5mms\"" Oct 30 17:52:22.311327 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311201 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hzdpx\"" Oct 30 17:52:22.311664 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311566 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6xmc6\"" Oct 30 17:52:22.311708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311660 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-signing-cabundle\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:22.311708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311670 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:52:22.311708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311662 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hhs6k\"" Oct 30 17:52:22.311708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.310992 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cxbhw\"" Oct 30 17:52:22.311708 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311664 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-config-volume\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:22.311920 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311737 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-x8qzn\"" Oct 30 17:52:22.311920 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311784 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lsd9l\"" Oct 30 17:52:22.311920 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311790 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dqbz4\"" Oct 30 17:52:22.311920 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311830 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ce04018-ba31-464c-8996-967b9100f174-agent-certs\") pod \"konnectivity-agent-vdlbp\" (UID: \"2ce04018-ba31-464c-8996-967b9100f174\") " pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:52:22.311920 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311091 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cgqqv\"" Oct 30 17:52:22.311920 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311859 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5kfdv\"" Oct 30 17:52:22.311920 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311882 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-ovnkube-script-lib\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.312188 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311973 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.312188 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.311999 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-env-overrides\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.312188 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.312039 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fzft4\"" Oct 30 17:52:22.312188 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.312056 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-stats-auth\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.312188 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.312067 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.312188 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.312172 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4c590f5-af8b-4994-8b03-7f5cee045767-ovn-node-metrics-cert\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.312372 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.312174 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-default-certificate\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.312372 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.312300 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.313267 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.313252 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d4c590f5-af8b-4994-8b03-7f5cee045767-ovnkube-config\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.313386 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.313368 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-original-pull-secret\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:22.313429 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.313410 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-serviceca\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:52:22.313554 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.313541 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-signing-key\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:22.322004 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.321912 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlkgn\" (UniqueName: \"kubernetes.io/projected/07abda71-7d42-43e6-9878-90ab7da7d9c4-kube-api-access-xlkgn\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.322004 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.321962 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xx66\" (UniqueName: \"kubernetes.io/projected/b6956a31-29a1-4cd7-bc8b-9cda43dc2e73-kube-api-access-4xx66\") pod \"kas-connection-checker-685895c7c9-2tjnd\" (UID: \"b6956a31-29a1-4cd7-bc8b-9cda43dc2e73\") " pod="kube-system/kas-connection-checker-685895c7c9-2tjnd" Oct 30 17:52:22.322004 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.321971 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmnr8\" (UniqueName: \"kubernetes.io/projected/5e94fb26-859a-48d2-a493-200543aa1881-kube-api-access-vmnr8\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.322159 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.322054 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdl6\" (UniqueName: \"kubernetes.io/projected/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-kube-api-access-bwdl6\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:22.322159 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.322123 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtx8j\" (UniqueName: \"kubernetes.io/projected/19bd958f-7d20-4da4-818f-06e23e01ab37-kube-api-access-rtx8j\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:22.322159 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.322144 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6kzk\" (UniqueName: \"kubernetes.io/projected/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-kube-api-access-r6kzk\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.322259 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.322180 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s2j6\" (UniqueName: \"kubernetes.io/projected/d5d1adae-1728-499a-9d16-ec24f1d69160-kube-api-access-4s2j6\") pod \"network-check-target-vqltd\" (UID: \"d5d1adae-1728-499a-9d16-ec24f1d69160\") " pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:22.322498 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.322455 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtbq\" (UniqueName: \"kubernetes.io/projected/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-kube-api-access-pmtbq\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:22.322538 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.322503 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-692s4\" (UniqueName: \"kubernetes.io/projected/d4c590f5-af8b-4994-8b03-7f5cee045767-kube-api-access-692s4\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.326410 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.326387 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv255\" (UniqueName: \"kubernetes.io/projected/3d610c80-f954-4f69-ab9a-e75c56c98652-kube-api-access-gv255\") pod \"migrator-66df5d496b-n52mh\" (UID: \"3d610c80-f954-4f69-ab9a-e75c56c98652\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n52mh" Oct 30 17:52:22.326490 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.326430 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtlqq\" (UniqueName: \"kubernetes.io/projected/06498a3b-c75b-4bf5-9fc7-beced86dbfc5-kube-api-access-gtlqq\") pod \"kas-connection-checker-685895c7c9-wjjnl\" (UID: \"06498a3b-c75b-4bf5-9fc7-beced86dbfc5\") " pod="kube-system/kas-connection-checker-685895c7c9-wjjnl" Oct 30 17:52:22.326596 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.326581 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pq9\" (UniqueName: \"kubernetes.io/projected/c1aa6b58-d1c3-40a0-873c-86b44c438fac-kube-api-access-j9pq9\") pod \"ingress-canary-t5n25\" (UID: \"c1aa6b58-d1c3-40a0-873c-86b44c438fac\") " pod="openshift-ingress-canary/ingress-canary-t5n25" Oct 30 17:52:22.327884 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.327864 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6mwh\" (UniqueName: \"kubernetes.io/projected/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-kube-api-access-j6mwh\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.328205 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.328188 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgj9b\" (UniqueName: \"kubernetes.io/projected/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-kube-api-access-mgj9b\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:52:22.330692 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.330671 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dxbs\" (UniqueName: \"kubernetes.io/projected/ba65a282-0992-49ce-b4bc-108c8f69016d-kube-api-access-2dxbs\") pod \"volume-data-source-validator-58c9f5f46d-hpv5m\" (UID: \"ba65a282-0992-49ce-b4bc-108c8f69016d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-hpv5m" Oct 30 17:52:22.330829 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.330811 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h54zm\" (UniqueName: \"kubernetes.io/projected/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-kube-api-access-h54zm\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:52:22.330890 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.330869 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf98n\" (UniqueName: \"kubernetes.io/projected/e4aa5e79-72a0-4b20-a639-ac9b61be4759-kube-api-access-nf98n\") pod \"service-ca-operator-544d749c8c-8kcxm\" (UID: \"e4aa5e79-72a0-4b20-a639-ac9b61be4759\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" Oct 30 17:52:22.334787 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.331337 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nslxb\" (UniqueName: \"kubernetes.io/projected/e45d20e6-c15c-4f25-9c13-d9cebc4775f0-kube-api-access-nslxb\") pod \"service-ca-5fffc97bdf-wql78\" (UID: \"e45d20e6-c15c-4f25-9c13-d9cebc4775f0\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wql78" Oct 30 17:52:22.334787 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.331431 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6bn\" (UniqueName: \"kubernetes.io/projected/4260cb2f-efee-4b45-9f73-a48757f3ce75-kube-api-access-zr6bn\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.334787 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.331780 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88w5h\" (UniqueName: \"kubernetes.io/projected/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-kube-api-access-88w5h\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:22.334787 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.331982 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr2r4\" (UniqueName: \"kubernetes.io/projected/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-api-access-sr2r4\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.334787 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.332205 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67r2z\" (UniqueName: \"kubernetes.io/projected/4fb42c05-fb07-4dae-975b-71d45dd52e03-kube-api-access-67r2z\") pod \"console-operator-7cd56b8dbc-5zcf9\" (UID: \"4fb42c05-fb07-4dae-975b-71d45dd52e03\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:22.334787 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.332841 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhvw\" (UniqueName: \"kubernetes.io/projected/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-kube-api-access-plhvw\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:52:22.334787 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.333353 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8v7l\" (UniqueName: \"kubernetes.io/projected/4ea379eb-5288-4f0f-90f4-712600a2fc01-kube-api-access-x8v7l\") pod \"periodic-gathering-x2wf2-g8lv5\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:52:22.334787 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.334152 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9756\" (UniqueName: \"kubernetes.io/projected/15f0fc8a-91c4-404a-91ec-b39914e83cae-kube-api-access-w9756\") pod \"insights-operator-7c4986d449-csr4p\" (UID: \"15f0fc8a-91c4-404a-91ec-b39914e83cae\") " pod="openshift-insights/insights-operator-7c4986d449-csr4p" Oct 30 17:52:22.404574 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404497 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-host\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.404574 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404541 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-lib-modules\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.404574 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404561 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-etc-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.404836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404622 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-host\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.404836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404669 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-etc-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.404836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404685 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-lib-modules\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.404836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404698 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-cni-bin\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.404836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404723 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-cni-bin\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.404836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404736 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-var-lib-kubelet\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.404836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404763 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-slash\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.404836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404785 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-run-netns\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.404836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404822 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-cni-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404844 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-conf-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404842 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-var-lib-kubelet\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404854 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-run-netns\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404843 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-slash\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404897 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-conf-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404904 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-cni-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404927 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-log-socket\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404964 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysctl-conf\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.404987 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cnibin\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405007 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-systemd-units\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405030 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405042 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-cnibin\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405075 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405073 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-systemd-units\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405073 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-log-socket\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405106 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysctl-conf\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405124 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405144 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-cni-bin\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405159 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-cnibin\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405174 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405209 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-cni-bin\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405232 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-socket-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405238 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405240 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-cnibin\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405250 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-registration-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405288 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-var-lib-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.405278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405292 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-socket-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.405299 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.905282801 +0000 UTC m=+1.862286637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : configmap references non-existent config key: service-ca.crt Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405334 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-registration-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405340 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-var-lib-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405344 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-sys-fs\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405368 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405372 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-sys-fs\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405407 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-systemd\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405427 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-cni-multus\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405442 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-kubelet-config\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.405453 4883 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405458 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-hosts-file\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405495 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-k8s-cni-cncf-io\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.405509 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls podName:9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.905496521 +0000 UTC m=+1.862500348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sw2jb" (UID: "9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405536 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-etc-kubernetes\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405543 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-kubelet-config\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405505 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97c8fb7c-e9a6-4eff-9a66-b70848c03a95-hosts-file\") pod \"node-resolver-xbxvk\" (UID: \"97c8fb7c-e9a6-4eff-9a66-b70848c03a95\") " pod="openshift-dns/node-resolver-xbxvk" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405566 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-cni-multus\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405571 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-host-slash\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405577 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-etc-kubernetes\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405539 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-k8s-cni-cncf-io\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405600 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-systemd\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405616 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-system-cni-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405624 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1e25bb5-1c5b-449e-a7f0-0deb010035ed-host-slash\") pod \"iptables-alerter-wbzt5\" (UID: \"d1e25bb5-1c5b-449e-a7f0-0deb010035ed\") " pod="openshift-network-operator/iptables-alerter-wbzt5" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405636 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405652 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-ovn\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405666 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405683 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-system-cni-dir\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405687 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-run\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405714 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-sys\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405722 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-ovn\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405726 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405740 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-hostroot\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.405751 4883 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405766 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-kubernetes\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405773 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-run\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405769 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-sys\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405796 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-hostroot\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.405815 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls podName:7cf0aaaf-7ca8-44b7-afad-829e4455e56a nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.905799487 +0000 UTC m=+1.862803320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls") pod "dns-default-6fncr" (UID: "7cf0aaaf-7ca8-44b7-afad-829e4455e56a") : secret "dns-default-metrics-tls" not found Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405800 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-systemd\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405830 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-kubernetes\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405834 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-systemd\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405848 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-run-ovn-kubernetes\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405867 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405885 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405906 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-device-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405930 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysconfig\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405929 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405974 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.405992 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-run-ovn-kubernetes\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406001 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406028 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-host\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406036 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d6f83f6-600c-4fa7-a20b-6a179c69bffc-device-dir\") pod \"aws-ebs-csi-driver-node-9h8s5\" (UID: \"9d6f83f6-600c-4fa7-a20b-6a179c69bffc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9h8s5" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406050 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-system-cni-dir\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406075 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-socket-dir-parent\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406100 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-netns\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.406109 4883 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406124 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-multus-certs\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.406157 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls podName:1f8536c2-cd14-4d88-ab1d-63a2f032aeb5 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.906143636 +0000 UTC m=+1.863147467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r7rxt" (UID: "1f8536c2-cd14-4d88-ab1d-63a2f032aeb5") : secret "samples-operator-tls" not found Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406170 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-multus-certs\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406174 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-modprobe-d\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406217 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysconfig\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406206 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-dbus\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406251 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysctl-d\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406275 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.406280 4883 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406297 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-cni-netd\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.406314 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs podName:19bd958f-7d20-4da4-818f-06e23e01ab37 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.906302139 +0000 UTC m=+1.863305966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs") pod "network-metrics-daemon-dl2lm" (UID: "19bd958f-7d20-4da4-818f-06e23e01ab37") : secret "metrics-daemon-secret" not found Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406330 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-cni-netd\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406341 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406367 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-crio-socket\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406389 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406412 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-kubelet\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406431 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-modprobe-d\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406439 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-kubelet\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406463 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-node-log\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406482 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44ce8b1c-6ab2-4f83-a1f6-183cc6cea104-dbus\") pod \"global-pull-secret-syncer-fglxs\" (UID: \"44ce8b1c-6ab2-4f83-a1f6-183cc6cea104\") " pod="kube-system/global-pull-secret-syncer-fglxs" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406528 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-node-log\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406561 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e94fb26-859a-48d2-a493-200543aa1881-etc-sysctl-d\") pod \"tuned-dpch6\" (UID: \"5e94fb26-859a-48d2-a493-200543aa1881\") " pod="openshift-cluster-node-tuning-operator/tuned-dpch6" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.406592 4883 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406597 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-run-openvswitch\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.406626 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:22.906614223 +0000 UTC m=+1.863618052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : secret "router-metrics-certs-default" not found Oct 30 17:52:22.406634 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406664 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-crio-socket\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.409358 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.406978 4883 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:52:22.409358 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.407173 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.409358 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.407262 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881-system-cni-dir\") pod \"multus-additional-cni-plugins-2sfml\" (UID: \"7e4a5fdf-7fb3-4e25-9dc7-7f48c6c06881\") " pod="openshift-multus/multus-additional-cni-plugins-2sfml" Oct 30 17:52:22.409358 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.407428 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-var-lib-kubelet\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.409358 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.407460 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c590f5-af8b-4994-8b03-7f5cee045767-host-kubelet\") pod \"ovnkube-node-k6zpc\" (UID: \"d4c590f5-af8b-4994-8b03-7f5cee045767\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:22.409358 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.407525 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0-host\") pod \"node-ca-7gt8q\" (UID: \"d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0\") " pod="openshift-image-registry/node-ca-7gt8q" Oct 30 17:52:22.409358 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.407580 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-multus-socket-dir-parent\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.409358 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.407618 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07abda71-7d42-43e6-9878-90ab7da7d9c4-host-run-netns\") pod \"multus-md4zx\" (UID: \"07abda71-7d42-43e6-9878-90ab7da7d9c4\") " pod="openshift-multus/multus-md4zx" Oct 30 17:52:22.409584 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.409432 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/867cb674-bd6b-4ca7-98a6-fa31bdaa132d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z2t26\" (UID: \"867cb674-bd6b-4ca7-98a6-fa31bdaa132d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:22.493410 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.493378 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" Oct 30 17:52:22.493555 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.493456 4883 scope.go:122] "RemoveContainer" containerID="9d4cc54e9eac0995064b052a1728805f57581f4f07a54b60d85f9eddfac462f5" Oct 30 17:52:22.828751 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.828719 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26"] Oct 30 17:52:22.834343 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:22.834301 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867cb674_bd6b_4ca7_98a6_fa31bdaa132d.slice/crio-ad80fb6e5bd2c690ab6a3f7f4f9eaf671c35e7fa8e361c87f8ed0bba964db447 WatchSource:0}: Error finding container ad80fb6e5bd2c690ab6a3f7f4f9eaf671c35e7fa8e361c87f8ed0bba964db447: Status 404 returned error can't find the container with id ad80fb6e5bd2c690ab6a3f7f4f9eaf671c35e7fa8e361c87f8ed0bba964db447 Oct 30 17:52:22.836380 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.836364 4883 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:52:22.909743 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.909674 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.909743 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.909712 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:22.909743 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.909745 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.909776 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.909799 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:22.909818 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.909860 4883 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.909912 4883 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.909925 4883 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.909955 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls podName:9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa nodeName:}" failed. No retries permitted until 2026-10-30 17:52:23.909913584 +0000 UTC m=+2.866917402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sw2jb" (UID: "9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.909913 4883 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.909987 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs podName:19bd958f-7d20-4da4-818f-06e23e01ab37 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:23.909971426 +0000 UTC m=+2.866975244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs") pod "network-metrics-daemon-dl2lm" (UID: "19bd958f-7d20-4da4-818f-06e23e01ab37") : secret "metrics-daemon-secret" not found Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.909916 4883 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.910002 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:23.909996327 +0000 UTC m=+2.867000140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : secret "router-metrics-certs-default" not found Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.910017 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:23.91000915 +0000 UTC m=+2.867012965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : configmap references non-existent config key: service-ca.crt Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.910033 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls podName:1f8536c2-cd14-4d88-ab1d-63a2f032aeb5 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:23.910024457 +0000 UTC m=+2.867028273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r7rxt" (UID: "1f8536c2-cd14-4d88-ab1d-63a2f032aeb5") : secret "samples-operator-tls" not found Oct 30 17:52:22.910065 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:22.910047 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls podName:7cf0aaaf-7ca8-44b7-afad-829e4455e56a nodeName:}" failed. No retries permitted until 2026-10-30 17:52:23.910039125 +0000 UTC m=+2.867042938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls") pod "dns-default-6fncr" (UID: "7cf0aaaf-7ca8-44b7-afad-829e4455e56a") : secret "dns-default-metrics-tls" not found Oct 30 17:52:23.240364 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.240241 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" event={"ID":"867cb674-bd6b-4ca7-98a6-fa31bdaa132d","Type":"ContainerStarted","Data":"ad80fb6e5bd2c690ab6a3f7f4f9eaf671c35e7fa8e361c87f8ed0bba964db447"} Oct 30 17:52:23.241758 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.241732 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 17:52:23.242148 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.242131 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/1.log" Oct 30 17:52:23.242194 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.242166 4883 generic.go:356] "Generic (PLEG): container finished" podID="4fb42c05-fb07-4dae-975b-71d45dd52e03" containerID="e55aeeae095977573ef42884c614b15f7bd8077ba3f3f276dd53c6764c3915fa" exitCode=255 Oct 30 17:52:23.242264 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.242240 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" event={"ID":"4fb42c05-fb07-4dae-975b-71d45dd52e03","Type":"ContainerDied","Data":"e55aeeae095977573ef42884c614b15f7bd8077ba3f3f276dd53c6764c3915fa"} Oct 30 17:52:23.242322 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.242309 4883 scope.go:122] "RemoveContainer" containerID="9d4cc54e9eac0995064b052a1728805f57581f4f07a54b60d85f9eddfac462f5" Oct 30 17:52:23.242918 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.242896 4883 scope.go:122] "RemoveContainer" containerID="e55aeeae095977573ef42884c614b15f7bd8077ba3f3f276dd53c6764c3915fa" Oct 30 17:52:23.243073 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:23.243056 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5zcf9_openshift-console-operator(4fb42c05-fb07-4dae-975b-71d45dd52e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" podUID="4fb42c05-fb07-4dae-975b-71d45dd52e03" Oct 30 17:52:23.373280 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.373254 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:23.920976 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.920919 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:23.921150 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.921002 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:23.921150 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.921057 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:23.921150 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.921096 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:23.921150 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:23.921134 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:25.921112456 +0000 UTC m=+4.878116286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : configmap references non-existent config key: service-ca.crt Oct 30 17:52:23.921374 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:23.921142 4883 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:23.921374 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:23.921223 4883 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:52:23.921374 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:23.921227 4883 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:52:23.921374 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:23.921235 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls podName:9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa nodeName:}" failed. No retries permitted until 2026-10-30 17:52:25.921219121 +0000 UTC m=+4.878222948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sw2jb" (UID: "9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:23.921374 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.921172 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:23.921374 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:23.921266 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs podName:19bd958f-7d20-4da4-818f-06e23e01ab37 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:25.921254112 +0000 UTC m=+4.878257959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs") pod "network-metrics-daemon-dl2lm" (UID: "19bd958f-7d20-4da4-818f-06e23e01ab37") : secret "metrics-daemon-secret" not found Oct 30 17:52:23.921374 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:23.921293 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls podName:1f8536c2-cd14-4d88-ab1d-63a2f032aeb5 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:25.921280512 +0000 UTC m=+4.878284332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r7rxt" (UID: "1f8536c2-cd14-4d88-ab1d-63a2f032aeb5") : secret "samples-operator-tls" not found Oct 30 17:52:23.921374 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.921309 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:23.921730 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:23.921457 4883 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:52:23.921730 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:23.921524 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs podName:4260cb2f-efee-4b45-9f73-a48757f3ce75 nodeName:}" failed. No retries permitted until 2026-10-30 17:52:25.921513362 +0000 UTC m=+4.878517193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs") pod "router-default-f6d947f75-ln5z8" (UID: "4260cb2f-efee-4b45-9f73-a48757f3ce75") : secret "router-metrics-certs-default" not found Oct 30 17:52:23.923677 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.923651 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cf0aaaf-7ca8-44b7-afad-829e4455e56a-metrics-tls\") pod \"dns-default-6fncr\" (UID: \"7cf0aaaf-7ca8-44b7-afad-829e4455e56a\") " pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:23.999614 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:23.999580 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:24.248165 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:24.248096 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 17:52:24.308208 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:24.308170 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6fncr"] Oct 30 17:52:24.316284 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:24.316252 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf0aaaf_7ca8_44b7_afad_829e4455e56a.slice/crio-b8ac430db6badc2d98a08b6b4cba12e90d95ee354a86183d83973d8aba64f38f WatchSource:0}: Error finding container b8ac430db6badc2d98a08b6b4cba12e90d95ee354a86183d83973d8aba64f38f: Status 404 returned error can't find the container with id b8ac430db6badc2d98a08b6b4cba12e90d95ee354a86183d83973d8aba64f38f Oct 30 17:52:25.253140 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.253106 4883 generic.go:356] "Generic (PLEG): container finished" podID="e4aa5e79-72a0-4b20-a639-ac9b61be4759" containerID="67529b2281e22b886000bff73777f92cd758f9f25ba7e958b6c4922b51b90669" exitCode=0 Oct 30 17:52:25.253545 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.253178 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" event={"ID":"e4aa5e79-72a0-4b20-a639-ac9b61be4759","Type":"ContainerDied","Data":"67529b2281e22b886000bff73777f92cd758f9f25ba7e958b6c4922b51b90669"} Oct 30 17:52:25.253545 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.253517 4883 scope.go:122] "RemoveContainer" containerID="67529b2281e22b886000bff73777f92cd758f9f25ba7e958b6c4922b51b90669" Oct 30 17:52:25.254833 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.254811 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z2t26" event={"ID":"867cb674-bd6b-4ca7-98a6-fa31bdaa132d","Type":"ContainerStarted","Data":"b0a3e5521dfb32be6c399916b59f1671e33eb2fc9de8febafd2fc116c7bc28e2"} Oct 30 17:52:25.256014 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.255995 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6fncr" event={"ID":"7cf0aaaf-7ca8-44b7-afad-829e4455e56a","Type":"ContainerStarted","Data":"b8ac430db6badc2d98a08b6b4cba12e90d95ee354a86183d83973d8aba64f38f"} Oct 30 17:52:25.940606 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.940581 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:25.940720 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.940615 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:25.940720 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.940635 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:25.940720 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.940693 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:25.940872 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.940729 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:25.940872 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:25.940845 4883 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:52:25.941141 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:25.940909 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls podName:9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa nodeName:}" failed. No retries permitted until 2026-10-30 17:52:29.940891275 +0000 UTC m=+8.897895107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sw2jb" (UID: "9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa") : secret "insights-runtime-extractor-tls" not found Oct 30 17:52:25.941439 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.941414 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4260cb2f-efee-4b45-9f73-a48757f3ce75-service-ca-bundle\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:25.943048 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.943026 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19bd958f-7d20-4da4-818f-06e23e01ab37-metrics-certs\") pod \"network-metrics-daemon-dl2lm\" (UID: \"19bd958f-7d20-4da4-818f-06e23e01ab37\") " pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:25.943118 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.943061 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8536c2-cd14-4d88-ab1d-63a2f032aeb5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r7rxt\" (UID: \"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:25.943274 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:25.943251 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4260cb2f-efee-4b45-9f73-a48757f3ce75-metrics-certs\") pod \"router-default-f6d947f75-ln5z8\" (UID: \"4260cb2f-efee-4b45-9f73-a48757f3ce75\") " pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:26.092576 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:26.092543 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" Oct 30 17:52:26.093714 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:26.093691 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:26.098336 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:26.098316 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dl2lm" Oct 30 17:52:26.261116 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:26.261060 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6fncr" event={"ID":"7cf0aaaf-7ca8-44b7-afad-829e4455e56a","Type":"ContainerStarted","Data":"20027c744a0f3eb1aab71f87c56c3a1741f566fb8d8f56b188b7b443a0ebb62a"} Oct 30 17:52:26.261545 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:26.261167 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6fncr" event={"ID":"7cf0aaaf-7ca8-44b7-afad-829e4455e56a","Type":"ContainerStarted","Data":"4e49333db1962e46794ffa19179d9269385387cad8bf071bbfa7d37a9d2c54bf"} Oct 30 17:52:26.262915 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:26.262891 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8kcxm" event={"ID":"e4aa5e79-72a0-4b20-a639-ac9b61be4759","Type":"ContainerStarted","Data":"20b61b749fe7396e8b896dda57606a09490a267954769756e1f80f466d664f2d"} Oct 30 17:52:26.390307 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:26.390238 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt"] Oct 30 17:52:26.408679 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:26.408650 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f6d947f75-ln5z8"] Oct 30 17:52:26.414881 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:26.414851 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4260cb2f_efee_4b45_9f73_a48757f3ce75.slice/crio-595eeb458a83c026dcc1642c0d2395b6cb3c8ca98060d8aa792287e2cec49def WatchSource:0}: Error finding container 595eeb458a83c026dcc1642c0d2395b6cb3c8ca98060d8aa792287e2cec49def: Status 404 returned error can't find the container with id 595eeb458a83c026dcc1642c0d2395b6cb3c8ca98060d8aa792287e2cec49def Oct 30 17:52:26.417091 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:26.416812 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dl2lm"] Oct 30 17:52:26.423947 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:26.423915 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bd958f_7d20_4da4_818f_06e23e01ab37.slice/crio-4f229508e0330c1303df5284ed9905bc504727ab87fe228798b8686e4066c0be WatchSource:0}: Error finding container 4f229508e0330c1303df5284ed9905bc504727ab87fe228798b8686e4066c0be: Status 404 returned error can't find the container with id 4f229508e0330c1303df5284ed9905bc504727ab87fe228798b8686e4066c0be Oct 30 17:52:26.981674 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:26.981631 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:27.274569 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:27.274483 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" event={"ID":"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5","Type":"ContainerStarted","Data":"18326268ae6bad2a7c4b07c52d2d738d698b59e00be27887d36f769306e02ce4"} Oct 30 17:52:27.276053 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:27.275915 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dl2lm" event={"ID":"19bd958f-7d20-4da4-818f-06e23e01ab37","Type":"ContainerStarted","Data":"4f229508e0330c1303df5284ed9905bc504727ab87fe228798b8686e4066c0be"} Oct 30 17:52:27.279098 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:27.279071 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f6d947f75-ln5z8" event={"ID":"4260cb2f-efee-4b45-9f73-a48757f3ce75","Type":"ContainerStarted","Data":"82099d64ec118f9dcdeb803e10b1f12bd8871e0bdc68a630b3d33a6a0a8ec40f"} Oct 30 17:52:27.279209 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:27.279118 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f6d947f75-ln5z8" event={"ID":"4260cb2f-efee-4b45-9f73-a48757f3ce75","Type":"ContainerStarted","Data":"595eeb458a83c026dcc1642c0d2395b6cb3c8ca98060d8aa792287e2cec49def"} Oct 30 17:52:28.094337 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:28.094304 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:28.096790 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:28.096767 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:28.283377 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:28.283285 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" event={"ID":"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5","Type":"ContainerStarted","Data":"54c3f001f29cedb77f94e1c960c4af0977363c1ef0a7565b288cf95e68ff0458"} Oct 30 17:52:28.283377 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:28.283348 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r7rxt" event={"ID":"1f8536c2-cd14-4d88-ab1d-63a2f032aeb5","Type":"ContainerStarted","Data":"95b28a18aab10473e71350aa264c34bbd201b1fa9d147a46c7bdb9c566c272c1"} Oct 30 17:52:28.285129 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:28.285096 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dl2lm" event={"ID":"19bd958f-7d20-4da4-818f-06e23e01ab37","Type":"ContainerStarted","Data":"15306c2aaaaa3fd0702da44dd70a1537aaae2fb23e8c4a0ebca873fbc7971583"} Oct 30 17:52:28.285239 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:28.285150 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dl2lm" event={"ID":"19bd958f-7d20-4da4-818f-06e23e01ab37","Type":"ContainerStarted","Data":"cfca23e980f8ee711df3cb959bb6201dac01e6df859c9a10022f0b24adcaf0c2"} Oct 30 17:52:29.071036 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:29.071004 4883 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:29.071385 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:29.071364 4883 scope.go:122] "RemoveContainer" containerID="e55aeeae095977573ef42884c614b15f7bd8077ba3f3f276dd53c6764c3915fa" Oct 30 17:52:29.071555 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:29.071536 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5zcf9_openshift-console-operator(4fb42c05-fb07-4dae-975b-71d45dd52e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" podUID="4fb42c05-fb07-4dae-975b-71d45dd52e03" Oct 30 17:52:29.288058 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:29.288027 4883 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:29.517901 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:29.517874 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:29.518129 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:29.518114 4883 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:29.974213 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:29.974173 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:29.976435 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:29.976411 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sw2jb\" (UID: \"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa\") " pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:30.001547 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:30.001512 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sw2jb" Oct 30 17:52:30.296183 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:30.296110 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sw2jb"] Oct 30 17:52:30.301615 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:30.301588 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7f9d5c_8fbe_439e_8ab5_3eb7891b01aa.slice/crio-ebfd7c77be173ddfcc5c91a4c6634000f7f690335d352e2ec0e122a459939713 WatchSource:0}: Error finding container ebfd7c77be173ddfcc5c91a4c6634000f7f690335d352e2ec0e122a459939713: Status 404 returned error can't find the container with id ebfd7c77be173ddfcc5c91a4c6634000f7f690335d352e2ec0e122a459939713 Oct 30 17:52:30.387381 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:30.387354 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:30.387522 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:30.387464 4883 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:52:30.388773 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:30.388751 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-f6d947f75-ln5z8" Oct 30 17:52:31.178566 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:31.178474 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:31.297237 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:31.297200 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sw2jb" event={"ID":"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa","Type":"ContainerStarted","Data":"bd1cf7c0acd541e25c77acfbf61f1bc04d3be3aab368a670e71e19ce61de6e5a"} Oct 30 17:52:31.297660 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:31.297270 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sw2jb" event={"ID":"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa","Type":"ContainerStarted","Data":"220dbdad9a3ac30936085dfcf7f7703d8490eded2b066047b2e773a926dcec32"} Oct 30 17:52:31.297660 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:31.297298 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sw2jb" event={"ID":"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa","Type":"ContainerStarted","Data":"ebfd7c77be173ddfcc5c91a4c6634000f7f690335d352e2ec0e122a459939713"} Oct 30 17:52:32.002056 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:32.002014 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:32.002504 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:32.002486 4883 scope.go:122] "RemoveContainer" containerID="e55aeeae095977573ef42884c614b15f7bd8077ba3f3f276dd53c6764c3915fa" Oct 30 17:52:32.002736 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:32.002697 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5zcf9_openshift-console-operator(4fb42c05-fb07-4dae-975b-71d45dd52e03)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" podUID="4fb42c05-fb07-4dae-975b-71d45dd52e03" Oct 30 17:52:32.306112 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:32.306031 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sw2jb" event={"ID":"9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa","Type":"ContainerStarted","Data":"7f671cd4cedca26086ade3a949329d348c6c0d0c487a4c2e5f0bcf65981abb53"} Oct 30 17:52:32.669366 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:32.669340 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6zpc" Oct 30 17:52:33.170928 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:33.170890 4883 scope.go:122] "RemoveContainer" containerID="e55aeeae095977573ef42884c614b15f7bd8077ba3f3f276dd53c6764c3915fa" Oct 30 17:52:33.310792 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:33.310767 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 17:52:33.311208 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:33.310889 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" event={"ID":"4fb42c05-fb07-4dae-975b-71d45dd52e03","Type":"ContainerStarted","Data":"0ee76d2bedb6479c6290f58cbced8dd6f7c74dc84cee26782c2f704e8aee97a4"} Oct 30 17:52:33.311321 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:33.311301 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:33.312245 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:33.312222 4883 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5zcf9 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Oct 30 17:52:33.312348 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:33.312262 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" podUID="4fb42c05-fb07-4dae-975b-71d45dd52e03" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Oct 30 17:52:34.318666 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:34.318638 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5zcf9" Oct 30 17:52:35.076807 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:52:35.076774 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:52:37.287786 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:37.287758 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6fncr" Oct 30 17:52:41.351823 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:41.351794 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vdlbp" Oct 30 17:52:46.028544 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.028502 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k8jm7"] Oct 30 17:52:46.061120 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.061086 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.084186 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.084159 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 17:52:46.089709 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.089684 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 17:52:46.089873 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.089726 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 17:52:46.089971 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.089947 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 17:52:46.089971 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.089949 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-58pv5\"" Oct 30 17:52:46.092869 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.092839 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.092974 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.092894 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-wtmp\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.093059 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.093008 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-root\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.093059 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.093027 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-textfile\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.093059 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.093046 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6c5r\" (UniqueName: \"kubernetes.io/projected/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-kube-api-access-x6c5r\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.093173 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.093066 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-sys\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.093173 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.093158 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.093256 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.093182 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-metrics-client-ca\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.093256 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.093201 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-tls\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.193772 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.193735 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-root\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.193772 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.193768 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-textfile\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194051 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.193788 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6c5r\" (UniqueName: \"kubernetes.io/projected/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-kube-api-access-x6c5r\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194051 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.193806 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-sys\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194051 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.193841 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194051 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.193879 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-metrics-client-ca\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194051 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.193894 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-sys\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194051 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.193906 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-tls\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194351 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.194009 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194351 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.194291 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-wtmp\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194351 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.193873 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-root\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194509 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.194215 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-textfile\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194509 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.194477 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-wtmp\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194509 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.194493 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-metrics-client-ca\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.194657 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.194590 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.196241 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.196222 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.196431 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.196413 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-node-exporter-tls\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.222100 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.222072 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6c5r\" (UniqueName: \"kubernetes.io/projected/bc2ac4e0-5b2c-4e3d-bffa-def481c62498-kube-api-access-x6c5r\") pod \"node-exporter-k8jm7\" (UID: \"bc2ac4e0-5b2c-4e3d-bffa-def481c62498\") " pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.411881 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:46.411843 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k8jm7" Oct 30 17:52:46.443969 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:46.443914 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2ac4e0_5b2c_4e3d_bffa_def481c62498.slice/crio-ab00ed3e7d72dad45b15d41a45ffab00a41a58530d90d77a588effe87c8d8f86 WatchSource:0}: Error finding container ab00ed3e7d72dad45b15d41a45ffab00a41a58530d90d77a588effe87c8d8f86: Status 404 returned error can't find the container with id ab00ed3e7d72dad45b15d41a45ffab00a41a58530d90d77a588effe87c8d8f86 Oct 30 17:52:47.363449 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:47.363405 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jm7" event={"ID":"bc2ac4e0-5b2c-4e3d-bffa-def481c62498","Type":"ContainerStarted","Data":"ab00ed3e7d72dad45b15d41a45ffab00a41a58530d90d77a588effe87c8d8f86"} Oct 30 17:52:48.368590 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:48.368554 4883 generic.go:356] "Generic (PLEG): container finished" podID="bc2ac4e0-5b2c-4e3d-bffa-def481c62498" containerID="9ed380955339d2368148bdc010f4b09a23b8cb638c42e1d6fe85f7205e22f4bb" exitCode=0 Oct 30 17:52:48.369022 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:48.368631 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jm7" event={"ID":"bc2ac4e0-5b2c-4e3d-bffa-def481c62498","Type":"ContainerDied","Data":"9ed380955339d2368148bdc010f4b09a23b8cb638c42e1d6fe85f7205e22f4bb"} Oct 30 17:52:49.373860 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:49.373824 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jm7" event={"ID":"bc2ac4e0-5b2c-4e3d-bffa-def481c62498","Type":"ContainerStarted","Data":"79622a196c5b403ea1176ad25f9afa0aba891bfc899c1a7d3fe3cf59c7bd5ab6"} Oct 30 17:52:49.374384 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:49.373880 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jm7" event={"ID":"bc2ac4e0-5b2c-4e3d-bffa-def481c62498","Type":"ContainerStarted","Data":"ca009b90445d5cfb83ec96d4e3105890460f538081ab730cad314ccc58b53b6a"} Oct 30 17:52:49.451656 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:49.451597 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k8jm7" podStartSLOduration=3.130972563 podStartE2EDuration="4.45158167s" totalImagesPullingTime="1.249255529s" totalInitContainerRuntime="71.353578ms" isStatefulPod=true podCreationTimestamp="2026-10-30 17:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:49.439454929 +0000 UTC m=+28.396458765" watchObservedRunningTime="2026-10-30 17:52:49.45158167 +0000 UTC m=+28.408585565" Oct 30 17:52:51.181145 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:51.181114 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vqltd" Oct 30 17:52:53.119335 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.119295 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 17:52:53.142442 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.142403 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.156017 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.155993 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 17:52:53.167173 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.167109 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 17:52:53.167173 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.167109 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 17:52:53.167173 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.167149 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 17:52:53.167173 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.167167 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 17:52:53.167173 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.167124 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 17:52:53.167557 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.167186 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ckcd9\"" Oct 30 17:52:53.167557 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.167114 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1j532o38mcikl\"" Oct 30 17:52:53.167557 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.167109 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 17:52:53.169619 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.169593 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 17:52:53.169728 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.169634 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 17:52:53.169767 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.169728 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 17:52:53.177584 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.177558 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 17:52:53.177584 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.177582 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 17:52:53.177766 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.177623 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 17:52:53.190286 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.190259 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 17:52:53.190430 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.190306 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 17:52:53.193927 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.193907 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 17:52:53.249270 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249228 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249270 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249270 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249533 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249295 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249533 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249396 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249533 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249423 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249533 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249450 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249533 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249519 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249714 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249632 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249714 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249664 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249714 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249696 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249823 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249731 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249823 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249760 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249823 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249786 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249999 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249831 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249999 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249889 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249999 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249926 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48kw5\" (UniqueName: \"kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-kube-api-access-48kw5\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.249999 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.249976 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.250194 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.250001 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.250194 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.250093 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.250194 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.250127 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.351408 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351370 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.351590 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351437 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.351590 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351475 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.351590 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351501 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.351590 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351525 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.351590 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351554 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.351590 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351580 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351607 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351638 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351668 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351699 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351735 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351759 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351784 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351812 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351841 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351868 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351900 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48kw5\" (UniqueName: \"kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-kube-api-access-48kw5\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351958 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.351987 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.352760 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.352728 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.353340 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.353311 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.353502 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.353475 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.353645 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.353564 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.354170 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.354150 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.354814 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.354736 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.355250 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.355206 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.355819 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.355793 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.356203 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.356184 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.357070 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.356994 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.357272 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.357246 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.358205 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.357554 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.358205 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.357764 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.358205 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.357972 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.358205 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.358011 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.358205 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.358161 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.358506 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.358398 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.358742 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.358715 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.358808 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.358778 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.378158 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.378097 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48kw5\" (UniqueName: \"kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-kube-api-access-48kw5\") pod \"prometheus-k8s-0\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.496352 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.496310 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:52:53.866337 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:53.866308 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 17:52:53.871503 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:52:53.871476 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e7b33c_28a1_4fda_8373_1fbf36f90c12.slice/crio-765fbf08f91438f4346ad74e99b7ea8f6a119e470a6c407b89a8258fdc4c7f7b WatchSource:0}: Error finding container 765fbf08f91438f4346ad74e99b7ea8f6a119e470a6c407b89a8258fdc4c7f7b: Status 404 returned error can't find the container with id 765fbf08f91438f4346ad74e99b7ea8f6a119e470a6c407b89a8258fdc4c7f7b Oct 30 17:52:54.392568 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:54.392531 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerStarted","Data":"765fbf08f91438f4346ad74e99b7ea8f6a119e470a6c407b89a8258fdc4c7f7b"} Oct 30 17:52:55.397739 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:55.397699 4883 generic.go:356] "Generic (PLEG): container finished" podID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerID="6a8cb20f11441a4e9c895320348a72e354de5c1fb64fbe2031209e99c63f9ed1" exitCode=0 Oct 30 17:52:55.398145 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:55.397783 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerDied","Data":"6a8cb20f11441a4e9c895320348a72e354de5c1fb64fbe2031209e99c63f9ed1"} Oct 30 17:52:58.413015 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:58.412920 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerStarted","Data":"e35a1a924bf3326bbe0756bc07ff359318a740bbad6d05efcc9f7168f74ebfe8"} Oct 30 17:52:58.413015 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:52:58.413007 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerStarted","Data":"d0aac656c0f38b5577189c9a620b7461ccb3cf0736ae8fb44ce9f09f1639d427"} Oct 30 17:53:01.427692 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:01.427658 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerStarted","Data":"93a83085e0e5bb55019dba1a288679022d44344da3983ff2e1597f19663059a8"} Oct 30 17:53:01.428207 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:01.427718 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerStarted","Data":"42fc1c9987e009dd9aa5e260f4198919aa73327f9a9f0021c8d1f2bc347d29cd"} Oct 30 17:53:01.428207 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:01.427744 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerStarted","Data":"5115f50e1bd1fd2fd29a00ab73d1576450f0ee7f35d710db7d5a7c248b0f7e24"} Oct 30 17:53:01.428207 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:01.427767 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerStarted","Data":"3616101612b562a9b1e8f0d936fffa7fc9c5560ee9fccede96960a04b353baba"} Oct 30 17:53:01.510482 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:01.510413 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.79535032 podStartE2EDuration="8.510398418s" totalImagesPullingTime="5.698557826s" totalInitContainerRuntime="16.490272ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:01.498345632 +0000 UTC m=+40.455349491" watchObservedRunningTime="2026-10-30 17:53:01.510398418 +0000 UTC m=+40.467402254" Oct 30 17:53:03.497067 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:03.496985 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:05.076821 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:53:05.076784 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:53:05.642083 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:05.642043 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:05.657655 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:05.657628 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:06.461830 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:06.461803 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:35.079247 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:53:35.079207 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:53:41.372523 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.372488 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 17:53:41.373079 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.372832 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="prometheus" containerID="cri-o://d0aac656c0f38b5577189c9a620b7461ccb3cf0736ae8fb44ce9f09f1639d427" gracePeriod=600 Oct 30 17:53:41.373079 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.372857 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="thanos-sidecar" containerID="cri-o://3616101612b562a9b1e8f0d936fffa7fc9c5560ee9fccede96960a04b353baba" gracePeriod=600 Oct 30 17:53:41.373079 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.372903 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="kube-rbac-proxy-thanos" containerID="cri-o://93a83085e0e5bb55019dba1a288679022d44344da3983ff2e1597f19663059a8" gracePeriod=600 Oct 30 17:53:41.373079 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.372895 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="config-reloader" containerID="cri-o://e35a1a924bf3326bbe0756bc07ff359318a740bbad6d05efcc9f7168f74ebfe8" gracePeriod=600 Oct 30 17:53:41.373079 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.372907 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="kube-rbac-proxy-web" containerID="cri-o://5115f50e1bd1fd2fd29a00ab73d1576450f0ee7f35d710db7d5a7c248b0f7e24" gracePeriod=600 Oct 30 17:53:41.373079 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.372855 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="kube-rbac-proxy" containerID="cri-o://42fc1c9987e009dd9aa5e260f4198919aa73327f9a9f0021c8d1f2bc347d29cd" gracePeriod=600 Oct 30 17:53:41.447014 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:53:41.446961 4883 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="d0aac656c0f38b5577189c9a620b7461ccb3cf0736ae8fb44ce9f09f1639d427" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 17:53:41.447116 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:53:41.447033 4883 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="d0aac656c0f38b5577189c9a620b7461ccb3cf0736ae8fb44ce9f09f1639d427" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 17:53:41.447116 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:53:41.447051 4883 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="d0aac656c0f38b5577189c9a620b7461ccb3cf0736ae8fb44ce9f09f1639d427" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 17:53:41.447116 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.447067 4883 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Oct 30 17:53:41.574985 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.574953 4883 generic.go:356] "Generic (PLEG): container finished" podID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerID="93a83085e0e5bb55019dba1a288679022d44344da3983ff2e1597f19663059a8" exitCode=0 Oct 30 17:53:41.574985 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.574982 4883 generic.go:356] "Generic (PLEG): container finished" podID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerID="42fc1c9987e009dd9aa5e260f4198919aa73327f9a9f0021c8d1f2bc347d29cd" exitCode=0 Oct 30 17:53:41.574985 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.574990 4883 generic.go:356] "Generic (PLEG): container finished" podID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerID="5115f50e1bd1fd2fd29a00ab73d1576450f0ee7f35d710db7d5a7c248b0f7e24" exitCode=0 Oct 30 17:53:41.575220 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.574998 4883 generic.go:356] "Generic (PLEG): container finished" podID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerID="3616101612b562a9b1e8f0d936fffa7fc9c5560ee9fccede96960a04b353baba" exitCode=0 Oct 30 17:53:41.575220 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.575005 4883 generic.go:356] "Generic (PLEG): container finished" podID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerID="e35a1a924bf3326bbe0756bc07ff359318a740bbad6d05efcc9f7168f74ebfe8" exitCode=0 Oct 30 17:53:41.575220 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.575012 4883 generic.go:356] "Generic (PLEG): container finished" podID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerID="d0aac656c0f38b5577189c9a620b7461ccb3cf0736ae8fb44ce9f09f1639d427" exitCode=0 Oct 30 17:53:41.575220 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.575019 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerDied","Data":"93a83085e0e5bb55019dba1a288679022d44344da3983ff2e1597f19663059a8"} Oct 30 17:53:41.575220 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.575105 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerDied","Data":"42fc1c9987e009dd9aa5e260f4198919aa73327f9a9f0021c8d1f2bc347d29cd"} Oct 30 17:53:41.575220 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.575133 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerDied","Data":"5115f50e1bd1fd2fd29a00ab73d1576450f0ee7f35d710db7d5a7c248b0f7e24"} Oct 30 17:53:41.575220 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.575161 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerDied","Data":"3616101612b562a9b1e8f0d936fffa7fc9c5560ee9fccede96960a04b353baba"} Oct 30 17:53:41.575220 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.575186 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerDied","Data":"e35a1a924bf3326bbe0756bc07ff359318a740bbad6d05efcc9f7168f74ebfe8"} Oct 30 17:53:41.575220 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.575208 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerDied","Data":"d0aac656c0f38b5577189c9a620b7461ccb3cf0736ae8fb44ce9f09f1639d427"} Oct 30 17:53:41.612793 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.612770 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:41.678178 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678082 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.678178 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678128 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.678178 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678154 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-db\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.678178 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678178 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-grpc-tls\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.678486 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678322 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-kube-api-access-48kw5\" (UniqueName: \"kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-kube-api-access-48kw5\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.678486 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678433 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-kubelet-serving-ca-bundle\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.678486 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678476 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-metrics-client-certs\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.678647 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678603 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-metrics-client-ca\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.678790 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678670 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config-out\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.678790 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678702 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-serving-certs-ca-bundle\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.678790 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678727 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-tls-assets\" (UniqueName: \"kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-tls-assets\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.678790 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678754 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-0\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678805 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-web-config\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-web-config\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678846 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.678972 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-db" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.679188 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-thanos-prometheus-http-client-file\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.679230 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-tls\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.679266 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-1\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.679306 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-2\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.679339 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-kube-rbac-proxy\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.679374 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-trusted-ca-bundle\") pod \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\" (UID: \"e4e7b33c-28a1-4fda-8373-1fbf36f90c12\") " Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.679405 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-serving-certs-ca-bundle" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.679659 4883 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-db\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.679683 4883 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.679993 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-trusted-ca-bundle" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.680342 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-2" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:41.681524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.680783 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-1" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:41.684046 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.681686 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-metrics-client-ca" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:41.684046 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.681823 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-kubelet-serving-ca-bundle" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:41.684046 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.682442 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-metrics-client-certs" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:41.684046 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.682507 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:41.684046 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.682771 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-grpc-tls" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:41.684046 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.682856 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-tls-assets" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:41.684046 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.683047 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-kube-api-access-48kw5" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "kube-api-access-48kw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:41.684046 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.683098 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-0" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:41.684046 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.683141 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-kube-rbac-proxy" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:41.684046 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.683889 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config-out" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:41.684548 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.684488 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-tls" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:41.684854 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.684820 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-thanos-prometheus-http-client-file" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:41.685837 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.685804 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-kube-rbac-proxy-web" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:41.686719 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.686699 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-thanos-sidecar-tls" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:41.697258 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.697224 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-web-config" pod "e4e7b33c-28a1-4fda-8373-1fbf36f90c12" (UID: "e4e7b33c-28a1-4fda-8373-1fbf36f90c12"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:41.780772 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780728 4883 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.780772 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780767 4883 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-metrics-client-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.780772 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780777 4883 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-configmap-metrics-client-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780787 4883 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config-out\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780796 4883 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-tls-assets\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780804 4883 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780813 4883 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-web-config\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780822 4883 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780831 4883 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780840 4883 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780850 4883 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780858 4883 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780866 4883 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-kube-rbac-proxy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780874 4883 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780883 4883 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780892 4883 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-config\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780899 4883 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-secret-grpc-tls\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:41.781042 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:41.780908 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48kw5\" (UniqueName: \"kubernetes.io/projected/e4e7b33c-28a1-4fda-8373-1fbf36f90c12-kube-api-access-48kw5\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:42.582115 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.582079 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4e7b33c-28a1-4fda-8373-1fbf36f90c12","Type":"ContainerDied","Data":"765fbf08f91438f4346ad74e99b7ea8f6a119e470a6c407b89a8258fdc4c7f7b"} Oct 30 17:53:42.582513 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.582172 4883 scope.go:122] "RemoveContainer" containerID="93a83085e0e5bb55019dba1a288679022d44344da3983ff2e1597f19663059a8" Oct 30 17:53:42.582513 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.582171 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.590081 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.590058 4883 scope.go:122] "RemoveContainer" containerID="42fc1c9987e009dd9aa5e260f4198919aa73327f9a9f0021c8d1f2bc347d29cd" Oct 30 17:53:42.596784 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.596764 4883 scope.go:122] "RemoveContainer" containerID="5115f50e1bd1fd2fd29a00ab73d1576450f0ee7f35d710db7d5a7c248b0f7e24" Oct 30 17:53:42.603446 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.603426 4883 scope.go:122] "RemoveContainer" containerID="3616101612b562a9b1e8f0d936fffa7fc9c5560ee9fccede96960a04b353baba" Oct 30 17:53:42.609925 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.609907 4883 scope.go:122] "RemoveContainer" containerID="e35a1a924bf3326bbe0756bc07ff359318a740bbad6d05efcc9f7168f74ebfe8" Oct 30 17:53:42.616309 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.616289 4883 scope.go:122] "RemoveContainer" containerID="d0aac656c0f38b5577189c9a620b7461ccb3cf0736ae8fb44ce9f09f1639d427" Oct 30 17:53:42.623066 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.623047 4883 scope.go:122] "RemoveContainer" containerID="6a8cb20f11441a4e9c895320348a72e354de5c1fb64fbe2031209e99c63f9ed1" Oct 30 17:53:42.647732 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.647676 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 17:53:42.659292 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.659270 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 17:53:42.682237 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682207 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 17:53:42.682604 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682585 4883 cpu_manager.go:414] "containerMap: removing container" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="config-reloader" Oct 30 17:53:42.682651 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682609 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="config-reloader" Oct 30 17:53:42.682651 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682633 4883 cpu_manager.go:414] "containerMap: removing container" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="init-config-reloader" Oct 30 17:53:42.682651 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682642 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="init-config-reloader" Oct 30 17:53:42.682734 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682653 4883 cpu_manager.go:414] "containerMap: removing container" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="thanos-sidecar" Oct 30 17:53:42.682734 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682662 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="thanos-sidecar" Oct 30 17:53:42.682734 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682685 4883 cpu_manager.go:414] "containerMap: removing container" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="kube-rbac-proxy-web" Oct 30 17:53:42.682734 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682693 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="kube-rbac-proxy-web" Oct 30 17:53:42.682734 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682726 4883 cpu_manager.go:414] "containerMap: removing container" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="kube-rbac-proxy-thanos" Oct 30 17:53:42.682734 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682733 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="kube-rbac-proxy-thanos" Oct 30 17:53:42.682912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682748 4883 cpu_manager.go:414] "containerMap: removing container" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="prometheus" Oct 30 17:53:42.682912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682753 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="prometheus" Oct 30 17:53:42.682912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682761 4883 cpu_manager.go:414] "containerMap: removing container" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="kube-rbac-proxy" Oct 30 17:53:42.682912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682765 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="kube-rbac-proxy" Oct 30 17:53:42.682912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682850 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="prometheus" Oct 30 17:53:42.682912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682863 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="kube-rbac-proxy" Oct 30 17:53:42.682912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682871 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="config-reloader" Oct 30 17:53:42.682912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682882 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="thanos-sidecar" Oct 30 17:53:42.682912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682894 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="kube-rbac-proxy-web" Oct 30 17:53:42.682912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.682906 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" containerName="kube-rbac-proxy-thanos" Oct 30 17:53:42.686508 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.686489 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.701678 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.701645 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 17:53:42.701888 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.701870 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 17:53:42.702033 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.701887 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 17:53:42.702033 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.701923 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 17:53:42.702033 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.701973 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 17:53:42.702033 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.702014 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 17:53:42.702033 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.702014 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 17:53:42.702268 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.702123 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 17:53:42.702268 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.702151 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 17:53:42.702268 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.702162 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 17:53:42.702268 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.702174 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ckcd9\"" Oct 30 17:53:42.702268 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.702209 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 17:53:42.702268 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.702260 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 17:53:42.702464 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.702375 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1j532o38mcikl\"" Oct 30 17:53:42.702464 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.702378 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 17:53:42.714692 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.714665 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 17:53:42.714776 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.714741 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 17:53:42.725550 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.725526 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 17:53:42.789325 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789290 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789325 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789325 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-web-config\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789545 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789342 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789545 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789410 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789545 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789464 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789545 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789485 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789545 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789508 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789545 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789533 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72c5389e-4e5d-49a0-9dea-8b460821f353-config-out\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789731 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789555 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789731 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789588 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789731 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789604 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789731 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789622 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789731 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789650 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789731 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789698 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789955 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789728 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789955 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789775 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789955 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789793 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvpk\" (UniqueName: \"kubernetes.io/projected/72c5389e-4e5d-49a0-9dea-8b460821f353-kube-api-access-pxvpk\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789955 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789811 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-config\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789955 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789829 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.789955 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.789915 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72c5389e-4e5d-49a0-9dea-8b460821f353-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891187 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891146 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72c5389e-4e5d-49a0-9dea-8b460821f353-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891378 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891203 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891378 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891240 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-web-config\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891378 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891269 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891378 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891306 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891378 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891340 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891378 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891369 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891648 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891408 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891648 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891454 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72c5389e-4e5d-49a0-9dea-8b460821f353-config-out\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891648 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891483 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891648 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891521 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891648 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891550 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891648 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891587 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891648 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891615 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891926 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891652 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891926 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891678 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891926 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891722 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891926 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891752 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvpk\" (UniqueName: \"kubernetes.io/projected/72c5389e-4e5d-49a0-9dea-8b460821f353-kube-api-access-pxvpk\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891926 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891787 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-config\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.891926 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.891815 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.896195 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.896166 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.896403 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.896252 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.896909 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.896877 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.898029 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.897976 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72c5389e-4e5d-49a0-9dea-8b460821f353-config-out\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.898126 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.898050 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.898191 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.898120 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.898262 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.898241 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.898324 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.898298 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-config\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.898566 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.898535 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.898979 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.898950 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.899072 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.898957 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.899072 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.898996 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.899185 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.899131 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.899981 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.899953 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/72c5389e-4e5d-49a0-9dea-8b460821f353-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.901049 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.901024 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-web-config\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.901167 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.901086 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.901273 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.901253 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.901371 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.901353 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72c5389e-4e5d-49a0-9dea-8b460821f353-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.901952 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.901909 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72c5389e-4e5d-49a0-9dea-8b460821f353-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:42.921834 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:42.921811 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvpk\" (UniqueName: \"kubernetes.io/projected/72c5389e-4e5d-49a0-9dea-8b460821f353-kube-api-access-pxvpk\") pod \"prometheus-k8s-0\" (UID: \"72c5389e-4e5d-49a0-9dea-8b460821f353\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:43.003108 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:43.003076 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:43.175164 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:43.175076 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e7b33c-28a1-4fda-8373-1fbf36f90c12" path="/var/lib/kubelet/pods/e4e7b33c-28a1-4fda-8373-1fbf36f90c12/volumes" Oct 30 17:53:43.339157 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:43.339119 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 17:53:43.344731 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:53:43.344702 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c5389e_4e5d_49a0_9dea_8b460821f353.slice/crio-ce289e7ced992e0f57149cf66f8e50265f83b2729774443a7c41366fe12c19db WatchSource:0}: Error finding container ce289e7ced992e0f57149cf66f8e50265f83b2729774443a7c41366fe12c19db: Status 404 returned error can't find the container with id ce289e7ced992e0f57149cf66f8e50265f83b2729774443a7c41366fe12c19db Oct 30 17:53:43.586833 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:43.586799 4883 generic.go:356] "Generic (PLEG): container finished" podID="72c5389e-4e5d-49a0-9dea-8b460821f353" containerID="1df32651b931a79512fa01278d5bc2b752bc4e3fa0fe8a0745d0fcabe6a13cea" exitCode=0 Oct 30 17:53:43.587303 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:43.586831 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72c5389e-4e5d-49a0-9dea-8b460821f353","Type":"ContainerDied","Data":"1df32651b931a79512fa01278d5bc2b752bc4e3fa0fe8a0745d0fcabe6a13cea"} Oct 30 17:53:43.587303 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:43.586902 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72c5389e-4e5d-49a0-9dea-8b460821f353","Type":"ContainerStarted","Data":"ce289e7ced992e0f57149cf66f8e50265f83b2729774443a7c41366fe12c19db"} Oct 30 17:53:44.594618 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:44.594581 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72c5389e-4e5d-49a0-9dea-8b460821f353","Type":"ContainerStarted","Data":"455cfa251cf919703d89c37acd7739e22d0343822478f7d6526a105dcb7475b9"} Oct 30 17:53:44.595136 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:44.594657 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72c5389e-4e5d-49a0-9dea-8b460821f353","Type":"ContainerStarted","Data":"d76247e4987bbd69cf14e6db9ec24188730265b57011615695c792234bcffdfa"} Oct 30 17:53:44.595136 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:44.594701 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72c5389e-4e5d-49a0-9dea-8b460821f353","Type":"ContainerStarted","Data":"861cc0cfd636a67a3fb0e24a08e6b8275711c42895b087ab6c48ada9b5524456"} Oct 30 17:53:44.595136 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:44.594744 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72c5389e-4e5d-49a0-9dea-8b460821f353","Type":"ContainerStarted","Data":"203c2e406c0caf886a12f9b601453f5eff1a23b9c0ceeb789b2fb1c4ead552af"} Oct 30 17:53:44.595136 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:44.594792 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72c5389e-4e5d-49a0-9dea-8b460821f353","Type":"ContainerStarted","Data":"88d0f1f4efd05693040a9099ffd74db10f1a45a984ac68a160493838e71b5232"} Oct 30 17:53:44.595136 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:44.594849 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"72c5389e-4e5d-49a0-9dea-8b460821f353","Type":"ContainerStarted","Data":"0e14a9cd240a5c066cfb5841b3c1f5cf7ef279f754f0722efca84ce02b77b384"} Oct 30 17:53:44.661712 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:44.661659 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.644833528 podStartE2EDuration="2.66164462s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.811092ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:44.660102062 +0000 UTC m=+83.617105897" watchObservedRunningTime="2026-10-30 17:53:44.66164462 +0000 UTC m=+83.618648456" Oct 30 17:53:48.003394 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:48.003344 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:53:49.614132 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:49.614101 4883 generic.go:356] "Generic (PLEG): container finished" podID="15f0fc8a-91c4-404a-91ec-b39914e83cae" containerID="b237fc030c991c4736cb3bfc76070dbde8658fa8c65ba4621c49e438e37429a2" exitCode=0 Oct 30 17:53:49.614431 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:49.614174 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-csr4p" event={"ID":"15f0fc8a-91c4-404a-91ec-b39914e83cae","Type":"ContainerDied","Data":"b237fc030c991c4736cb3bfc76070dbde8658fa8c65ba4621c49e438e37429a2"} Oct 30 17:53:49.614489 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:49.614476 4883 scope.go:122] "RemoveContainer" containerID="b237fc030c991c4736cb3bfc76070dbde8658fa8c65ba4621c49e438e37429a2" Oct 30 17:53:50.124050 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.124016 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-x2wf2-g8lv5"] Oct 30 17:53:50.124292 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.124266 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" podUID="4ea379eb-5288-4f0f-90f4-712600a2fc01" containerName="insights-gathering" containerID="cri-o://452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c" gracePeriod=30 Oct 30 17:53:50.319138 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.319100 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-sv7x6-6h4f9"] Oct 30 17:53:50.321508 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.321486 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.344010 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.343945 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sv7x6-6h4f9"] Oct 30 17:53:50.366202 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.366180 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:53:50.456076 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.455995 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-service-ca-bundle\") pod \"4ea379eb-5288-4f0f-90f4-712600a2fc01\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " Oct 30 17:53:50.456076 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456064 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-trusted-ca-bundle\") pod \"4ea379eb-5288-4f0f-90f4-712600a2fc01\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " Oct 30 17:53:50.456279 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456096 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea379eb-5288-4f0f-90f4-712600a2fc01-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ea379eb-5288-4f0f-90f4-712600a2fc01-archives-path\") pod \"4ea379eb-5288-4f0f-90f4-712600a2fc01\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " Oct 30 17:53:50.456279 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456124 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea379eb-5288-4f0f-90f4-712600a2fc01-kube-api-access-x8v7l\" (UniqueName: \"kubernetes.io/projected/4ea379eb-5288-4f0f-90f4-712600a2fc01-kube-api-access-x8v7l\") pod \"4ea379eb-5288-4f0f-90f4-712600a2fc01\" (UID: \"4ea379eb-5288-4f0f-90f4-712600a2fc01\") " Oct 30 17:53:50.456279 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456223 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b5afb42c-f785-4b8f-a6ca-2236d724d8af-archives-path\") pod \"periodic-gathering-sv7x6-6h4f9\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.456279 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456274 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-service-ca-bundle\") pod \"periodic-gathering-sv7x6-6h4f9\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.456497 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456404 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-service-ca-bundle" pod "4ea379eb-5288-4f0f-90f4-712600a2fc01" (UID: "4ea379eb-5288-4f0f-90f4-712600a2fc01"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:50.456497 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456428 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-trusted-ca-bundle\") pod \"periodic-gathering-sv7x6-6h4f9\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.456497 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456437 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea379eb-5288-4f0f-90f4-712600a2fc01-archives-path" pod "4ea379eb-5288-4f0f-90f4-712600a2fc01" (UID: "4ea379eb-5288-4f0f-90f4-712600a2fc01"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:50.456612 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456497 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-trusted-ca-bundle" pod "4ea379eb-5288-4f0f-90f4-712600a2fc01" (UID: "4ea379eb-5288-4f0f-90f4-712600a2fc01"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:50.456612 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456505 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9kcb\" (UniqueName: \"kubernetes.io/projected/b5afb42c-f785-4b8f-a6ca-2236d724d8af-kube-api-access-m9kcb\") pod \"periodic-gathering-sv7x6-6h4f9\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.456612 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456560 4883 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-service-ca-bundle\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:50.456612 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456572 4883 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ea379eb-5288-4f0f-90f4-712600a2fc01-trusted-ca-bundle\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:50.456612 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.456596 4883 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ea379eb-5288-4f0f-90f4-712600a2fc01-archives-path\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:50.458472 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.458453 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea379eb-5288-4f0f-90f4-712600a2fc01-kube-api-access-x8v7l" pod "4ea379eb-5288-4f0f-90f4-712600a2fc01" (UID: "4ea379eb-5288-4f0f-90f4-712600a2fc01"). InnerVolumeSpecName "kube-api-access-x8v7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:50.557894 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.557854 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-trusted-ca-bundle\") pod \"periodic-gathering-sv7x6-6h4f9\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.557894 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.557897 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9kcb\" (UniqueName: \"kubernetes.io/projected/b5afb42c-f785-4b8f-a6ca-2236d724d8af-kube-api-access-m9kcb\") pod \"periodic-gathering-sv7x6-6h4f9\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.558166 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.557924 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b5afb42c-f785-4b8f-a6ca-2236d724d8af-archives-path\") pod \"periodic-gathering-sv7x6-6h4f9\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.558166 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.557980 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-service-ca-bundle\") pod \"periodic-gathering-sv7x6-6h4f9\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.558166 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.558126 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8v7l\" (UniqueName: \"kubernetes.io/projected/4ea379eb-5288-4f0f-90f4-712600a2fc01-kube-api-access-x8v7l\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:53:50.558388 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.558360 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b5afb42c-f785-4b8f-a6ca-2236d724d8af-archives-path\") pod \"periodic-gathering-sv7x6-6h4f9\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.558706 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.558685 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-trusted-ca-bundle\") pod \"periodic-gathering-sv7x6-6h4f9\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.558755 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.558719 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-service-ca-bundle\") pod \"periodic-gathering-sv7x6-6h4f9\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.581263 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.581239 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9kcb\" (UniqueName: \"kubernetes.io/projected/b5afb42c-f785-4b8f-a6ca-2236d724d8af-kube-api-access-m9kcb\") pod \"periodic-gathering-sv7x6-6h4f9\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.619569 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.619537 4883 generic.go:356] "Generic (PLEG): container finished" podID="4ea379eb-5288-4f0f-90f4-712600a2fc01" containerID="452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c" exitCode=2 Oct 30 17:53:50.619930 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.619582 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" event={"ID":"4ea379eb-5288-4f0f-90f4-712600a2fc01","Type":"ContainerDied","Data":"452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c"} Oct 30 17:53:50.619930 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.619633 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" event={"ID":"4ea379eb-5288-4f0f-90f4-712600a2fc01","Type":"ContainerDied","Data":"afde3c89beda09c91adcc60ebd6d6823c774cdc8ae9d4a0911a77f5f21d8c65a"} Oct 30 17:53:50.619930 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.619590 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x2wf2-g8lv5" Oct 30 17:53:50.619930 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.619660 4883 scope.go:122] "RemoveContainer" containerID="452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c" Oct 30 17:53:50.621613 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.621586 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-csr4p" event={"ID":"15f0fc8a-91c4-404a-91ec-b39914e83cae","Type":"ContainerStarted","Data":"c6219661d59bcc1e0e117f748199394bfee99ca867e6562a3665dc8ce6dae744"} Oct 30 17:53:50.627604 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.627559 4883 scope.go:122] "RemoveContainer" containerID="452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c" Oct 30 17:53:50.627847 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:53:50.627828 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c\": container with ID starting with 452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c not found: ID does not exist" containerID="452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c" Oct 30 17:53:50.627913 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.627854 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c"} err="failed to get container status \"452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c\": rpc error: code = NotFound desc = could not find container \"452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c\": container with ID starting with 452d8e25d6a64d199ec7124e2e75610023c8127ba01df713aa3be469549e9e1c not found: ID does not exist" Oct 30 17:53:50.632023 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.632006 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:53:50.670087 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.670041 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-x2wf2-g8lv5"] Oct 30 17:53:50.673647 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.673619 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-x2wf2-g8lv5"] Oct 30 17:53:50.929777 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:50.929741 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sv7x6-6h4f9"] Oct 30 17:53:50.935245 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:53:50.935221 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5afb42c_f785_4b8f_a6ca_2236d724d8af.slice/crio-793328e8acf11e9e91acc2dec57335894d4d534671b1ae0725de558a01b490d6 WatchSource:0}: Error finding container 793328e8acf11e9e91acc2dec57335894d4d534671b1ae0725de558a01b490d6: Status 404 returned error can't find the container with id 793328e8acf11e9e91acc2dec57335894d4d534671b1ae0725de558a01b490d6 Oct 30 17:53:51.174928 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:51.174899 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea379eb-5288-4f0f-90f4-712600a2fc01" path="/var/lib/kubelet/pods/4ea379eb-5288-4f0f-90f4-712600a2fc01/volumes" Oct 30 17:53:51.627561 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:51.627531 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" event={"ID":"b5afb42c-f785-4b8f-a6ca-2236d724d8af","Type":"ContainerStarted","Data":"74625b21354ad350598b67ace946fba29df2427ff1dab8d0e8243ec41eaf2f02"} Oct 30 17:53:51.627964 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:51.627577 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" event={"ID":"b5afb42c-f785-4b8f-a6ca-2236d724d8af","Type":"ContainerStarted","Data":"793328e8acf11e9e91acc2dec57335894d4d534671b1ae0725de558a01b490d6"} Oct 30 17:53:51.667091 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:53:51.667036 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" podStartSLOduration=1.667021046 podStartE2EDuration="1.667021046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:51.666418154 +0000 UTC m=+90.623421991" watchObservedRunningTime="2026-10-30 17:53:51.667021046 +0000 UTC m=+90.624024881" Oct 30 17:54:05.076992 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:54:05.076926 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:54:35.077035 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:54:35.076954 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:54:43.003413 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:54:43.003371 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:54:43.019183 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:54:43.019154 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:54:43.835952 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:54:43.835914 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 17:55:05.076822 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:55:05.076771 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:55:21.953595 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:21.953558 4883 generic.go:356] "Generic (PLEG): container finished" podID="b5afb42c-f785-4b8f-a6ca-2236d724d8af" containerID="74625b21354ad350598b67ace946fba29df2427ff1dab8d0e8243ec41eaf2f02" exitCode=0 Oct 30 17:55:21.954121 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:21.953639 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" event={"ID":"b5afb42c-f785-4b8f-a6ca-2236d724d8af","Type":"ContainerDied","Data":"74625b21354ad350598b67ace946fba29df2427ff1dab8d0e8243ec41eaf2f02"} Oct 30 17:55:23.090170 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.090147 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:55:23.125542 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.125513 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-trusted-ca-bundle\") pod \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " Oct 30 17:55:23.125681 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.125600 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-service-ca-bundle\") pod \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " Oct 30 17:55:23.125734 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.125704 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5afb42c-f785-4b8f-a6ca-2236d724d8af-kube-api-access-m9kcb\" (UniqueName: \"kubernetes.io/projected/b5afb42c-f785-4b8f-a6ca-2236d724d8af-kube-api-access-m9kcb\") pod \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " Oct 30 17:55:23.125795 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.125779 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5afb42c-f785-4b8f-a6ca-2236d724d8af-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b5afb42c-f785-4b8f-a6ca-2236d724d8af-archives-path\") pod \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\" (UID: \"b5afb42c-f785-4b8f-a6ca-2236d724d8af\") " Oct 30 17:55:23.125967 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.125926 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-service-ca-bundle" pod "b5afb42c-f785-4b8f-a6ca-2236d724d8af" (UID: "b5afb42c-f785-4b8f-a6ca-2236d724d8af"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:55:23.126028 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.125976 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-trusted-ca-bundle" pod "b5afb42c-f785-4b8f-a6ca-2236d724d8af" (UID: "b5afb42c-f785-4b8f-a6ca-2236d724d8af"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:55:23.126156 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.126134 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5afb42c-f785-4b8f-a6ca-2236d724d8af-archives-path" pod "b5afb42c-f785-4b8f-a6ca-2236d724d8af" (UID: "b5afb42c-f785-4b8f-a6ca-2236d724d8af"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:23.127771 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.127748 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5afb42c-f785-4b8f-a6ca-2236d724d8af-kube-api-access-m9kcb" pod "b5afb42c-f785-4b8f-a6ca-2236d724d8af" (UID: "b5afb42c-f785-4b8f-a6ca-2236d724d8af"). InnerVolumeSpecName "kube-api-access-m9kcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:23.227055 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.226959 4883 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b5afb42c-f785-4b8f-a6ca-2236d724d8af-archives-path\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:55:23.227055 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.226997 4883 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-trusted-ca-bundle\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:55:23.227055 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.227010 4883 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5afb42c-f785-4b8f-a6ca-2236d724d8af-service-ca-bundle\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:55:23.227055 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.227023 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9kcb\" (UniqueName: \"kubernetes.io/projected/b5afb42c-f785-4b8f-a6ca-2236d724d8af-kube-api-access-m9kcb\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:55:23.961056 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.961013 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" event={"ID":"b5afb42c-f785-4b8f-a6ca-2236d724d8af","Type":"ContainerDied","Data":"793328e8acf11e9e91acc2dec57335894d4d534671b1ae0725de558a01b490d6"} Oct 30 17:55:23.961219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.961063 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="793328e8acf11e9e91acc2dec57335894d4d534671b1ae0725de558a01b490d6" Oct 30 17:55:23.961219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:55:23.961032 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sv7x6-6h4f9" Oct 30 17:55:35.077190 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:55:35.077154 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:56:05.075803 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:56:05.075725 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:56:32.235785 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.235752 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9"] Oct 30 17:56:32.236197 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.236068 4883 cpu_manager.go:414] "containerMap: removing container" podUID="4ea379eb-5288-4f0f-90f4-712600a2fc01" containerName="insights-gathering" Oct 30 17:56:32.236197 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.236080 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea379eb-5288-4f0f-90f4-712600a2fc01" containerName="insights-gathering" Oct 30 17:56:32.236197 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.236106 4883 cpu_manager.go:414] "containerMap: removing container" podUID="b5afb42c-f785-4b8f-a6ca-2236d724d8af" containerName="insights-gathering" Oct 30 17:56:32.236197 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.236111 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5afb42c-f785-4b8f-a6ca-2236d724d8af" containerName="insights-gathering" Oct 30 17:56:32.236197 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.236159 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5afb42c-f785-4b8f-a6ca-2236d724d8af" containerName="insights-gathering" Oct 30 17:56:32.236197 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.236169 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea379eb-5288-4f0f-90f4-712600a2fc01" containerName="insights-gathering" Oct 30 17:56:32.239240 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.239224 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.274470 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.274441 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd28ee13-7aff-4e16-aa2e-5813174f58cb-metrics-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.274470 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.274472 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwh22\" (UniqueName: \"kubernetes.io/projected/fd28ee13-7aff-4e16-aa2e-5813174f58cb-kube-api-access-bwh22\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.274638 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.274494 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd28ee13-7aff-4e16-aa2e-5813174f58cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.274638 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.274588 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd28ee13-7aff-4e16-aa2e-5813174f58cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.274638 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.274632 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd28ee13-7aff-4e16-aa2e-5813174f58cb-manifests\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.274734 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.274685 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd28ee13-7aff-4e16-aa2e-5813174f58cb-tmp\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.282028 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.281980 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9"] Oct 30 17:56:32.321975 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.321927 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 17:56:32.321975 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.321968 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 17:56:32.322363 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.322330 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 17:56:32.322531 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.322518 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 17:56:32.322637 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.322622 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 17:56:32.322725 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.322714 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-v2gs7\"" Oct 30 17:56:32.375387 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.375344 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd28ee13-7aff-4e16-aa2e-5813174f58cb-metrics-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.375387 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.375387 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwh22\" (UniqueName: \"kubernetes.io/projected/fd28ee13-7aff-4e16-aa2e-5813174f58cb-kube-api-access-bwh22\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.375629 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.375420 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd28ee13-7aff-4e16-aa2e-5813174f58cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.375629 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.375454 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd28ee13-7aff-4e16-aa2e-5813174f58cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.375629 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.375499 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd28ee13-7aff-4e16-aa2e-5813174f58cb-manifests\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.375629 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.375536 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd28ee13-7aff-4e16-aa2e-5813174f58cb-tmp\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.375959 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.375918 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd28ee13-7aff-4e16-aa2e-5813174f58cb-manifests\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.377759 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.377730 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd28ee13-7aff-4e16-aa2e-5813174f58cb-tmp\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.377917 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.377897 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd28ee13-7aff-4e16-aa2e-5813174f58cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.378246 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.378228 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd28ee13-7aff-4e16-aa2e-5813174f58cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.378305 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.378285 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd28ee13-7aff-4e16-aa2e-5813174f58cb-metrics-cert\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.400798 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.400772 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwh22\" (UniqueName: \"kubernetes.io/projected/fd28ee13-7aff-4e16-aa2e-5813174f58cb-kube-api-access-bwh22\") pod \"opendatahub-operator-controller-manager-64bcb8d955-zr5b9\" (UID: \"fd28ee13-7aff-4e16-aa2e-5813174f58cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.553058 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.552975 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:32.920907 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:32.920876 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9"] Oct 30 17:56:32.924236 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:56:32.924208 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd28ee13_7aff_4e16_aa2e_5813174f58cb.slice/crio-2540d8b18e22a17f54685a0c83c1534d02e1096842f37f07696f5c5d96cbda0a WatchSource:0}: Error finding container 2540d8b18e22a17f54685a0c83c1534d02e1096842f37f07696f5c5d96cbda0a: Status 404 returned error can't find the container with id 2540d8b18e22a17f54685a0c83c1534d02e1096842f37f07696f5c5d96cbda0a Oct 30 17:56:33.205981 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:33.205890 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" event={"ID":"fd28ee13-7aff-4e16-aa2e-5813174f58cb","Type":"ContainerStarted","Data":"2540d8b18e22a17f54685a0c83c1534d02e1096842f37f07696f5c5d96cbda0a"} Oct 30 17:56:35.078905 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:56:35.078868 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:56:36.222092 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:36.222058 4883 generic.go:356] "Generic (PLEG): container finished" podID="fd28ee13-7aff-4e16-aa2e-5813174f58cb" containerID="a937033d67f81e1ea9996084581f0dc37b699fcd77333fef98cec12008532ea9" exitCode=0 Oct 30 17:56:36.222605 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:36.222093 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" event={"ID":"fd28ee13-7aff-4e16-aa2e-5813174f58cb","Type":"ContainerDied","Data":"a937033d67f81e1ea9996084581f0dc37b699fcd77333fef98cec12008532ea9"} Oct 30 17:56:37.227679 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:37.227645 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" event={"ID":"fd28ee13-7aff-4e16-aa2e-5813174f58cb","Type":"ContainerStarted","Data":"9bef7e544f3a9f3a1ce2a016c317013b007449f9a9a7454e6e766e8f092b75f0"} Oct 30 17:56:37.228169 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:37.227769 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:37.343181 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:37.343128 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" podStartSLOduration=1.939081997 podStartE2EDuration="5.343115387s" totalImagesPullingTime="2.99682374s" totalInitContainerRuntime="407.20965ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:37.343019421 +0000 UTC m=+256.300023256" watchObservedRunningTime="2026-10-30 17:56:37.343115387 +0000 UTC m=+256.300119221" Oct 30 17:56:48.233602 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:48.233563 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" Oct 30 17:56:51.347712 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.347676 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm"] Oct 30 17:56:51.352861 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.352837 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm" Oct 30 17:56:51.371054 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.371029 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm"] Oct 30 17:56:51.382747 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.382725 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:51.383092 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.383071 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:51.383181 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.383101 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-sdq86\"" Oct 30 17:56:51.443947 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.443909 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f1b8243-22f6-4780-bd31-09e647f0833d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm\" (UID: \"0f1b8243-22f6-4780-bd31-09e647f0833d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm" Oct 30 17:56:51.444119 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.443972 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7fw4\" (UniqueName: \"kubernetes.io/projected/0f1b8243-22f6-4780-bd31-09e647f0833d-kube-api-access-r7fw4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm\" (UID: \"0f1b8243-22f6-4780-bd31-09e647f0833d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm" Oct 30 17:56:51.545032 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.545002 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f1b8243-22f6-4780-bd31-09e647f0833d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm\" (UID: \"0f1b8243-22f6-4780-bd31-09e647f0833d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm" Oct 30 17:56:51.545194 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.545038 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7fw4\" (UniqueName: \"kubernetes.io/projected/0f1b8243-22f6-4780-bd31-09e647f0833d-kube-api-access-r7fw4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm\" (UID: \"0f1b8243-22f6-4780-bd31-09e647f0833d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm" Oct 30 17:56:51.545376 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.545356 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f1b8243-22f6-4780-bd31-09e647f0833d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm\" (UID: \"0f1b8243-22f6-4780-bd31-09e647f0833d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm" Oct 30 17:56:51.569773 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.569736 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7fw4\" (UniqueName: \"kubernetes.io/projected/0f1b8243-22f6-4780-bd31-09e647f0833d-kube-api-access-r7fw4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm\" (UID: \"0f1b8243-22f6-4780-bd31-09e647f0833d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm" Oct 30 17:56:51.662910 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:51.662809 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm" Oct 30 17:56:52.011588 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:52.011510 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm"] Oct 30 17:56:52.015124 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:56:52.015096 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1b8243_22f6_4780_bd31_09e647f0833d.slice/crio-3aeffdd7707c34349ffa9dd6246bc2bc6350f9991ddec63d83a7c9b63d1e33c6 WatchSource:0}: Error finding container 3aeffdd7707c34349ffa9dd6246bc2bc6350f9991ddec63d83a7c9b63d1e33c6: Status 404 returned error can't find the container with id 3aeffdd7707c34349ffa9dd6246bc2bc6350f9991ddec63d83a7c9b63d1e33c6 Oct 30 17:56:52.286474 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:52.286384 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm" event={"ID":"0f1b8243-22f6-4780-bd31-09e647f0833d","Type":"ContainerStarted","Data":"3aeffdd7707c34349ffa9dd6246bc2bc6350f9991ddec63d83a7c9b63d1e33c6"} Oct 30 17:56:55.301674 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:55.301636 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm" event={"ID":"0f1b8243-22f6-4780-bd31-09e647f0833d","Type":"ContainerStarted","Data":"2b710dc4f3f5b95403d51a63fffe2d79a859c148b57c820dcbf6d382eed04bdc"} Oct 30 17:56:55.376628 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:55.376573 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mr5cm" podStartSLOduration=1.79970409 podStartE2EDuration="4.376557092s" totalImagesPullingTime="2.576853002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:55.361836099 +0000 UTC m=+274.318839934" watchObservedRunningTime="2026-10-30 17:56:55.376557092 +0000 UTC m=+274.333560925" Oct 30 17:56:57.739735 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:57.739699 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9"] Oct 30 17:56:57.742153 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:57.742131 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" Oct 30 17:56:57.758630 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:57.758607 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 17:56:57.758744 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:57.758698 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 17:56:57.765204 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:57.765176 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9"] Oct 30 17:56:57.799434 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:57.799400 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50a26671-57c2-4781-9b78-211ea23c0318-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-m2fn9\" (UID: \"50a26671-57c2-4781-9b78-211ea23c0318\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" Oct 30 17:56:57.799598 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:57.799445 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzngq\" (UniqueName: \"kubernetes.io/projected/50a26671-57c2-4781-9b78-211ea23c0318-kube-api-access-tzngq\") pod \"cert-manager-webhook-5dbbbf75cc-m2fn9\" (UID: \"50a26671-57c2-4781-9b78-211ea23c0318\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" Oct 30 17:56:57.836128 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:57.836093 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rlj4w\"" Oct 30 17:56:57.900655 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:57.900620 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50a26671-57c2-4781-9b78-211ea23c0318-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-m2fn9\" (UID: \"50a26671-57c2-4781-9b78-211ea23c0318\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" Oct 30 17:56:57.900860 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:57.900664 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzngq\" (UniqueName: \"kubernetes.io/projected/50a26671-57c2-4781-9b78-211ea23c0318-kube-api-access-tzngq\") pod \"cert-manager-webhook-5dbbbf75cc-m2fn9\" (UID: \"50a26671-57c2-4781-9b78-211ea23c0318\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" Oct 30 17:56:57.927017 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:57.926699 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzngq\" (UniqueName: \"kubernetes.io/projected/50a26671-57c2-4781-9b78-211ea23c0318-kube-api-access-tzngq\") pod \"cert-manager-webhook-5dbbbf75cc-m2fn9\" (UID: \"50a26671-57c2-4781-9b78-211ea23c0318\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" Oct 30 17:56:57.929516 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:57.929491 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50a26671-57c2-4781-9b78-211ea23c0318-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-m2fn9\" (UID: \"50a26671-57c2-4781-9b78-211ea23c0318\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" Oct 30 17:56:58.051036 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.050957 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg"] Oct 30 17:56:58.053200 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.053181 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg" Oct 30 17:56:58.067793 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.067767 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" Oct 30 17:56:58.069267 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.069242 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 17:56:58.069401 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.069385 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"mcp-gateway-controller-dockercfg-wmkn2\"" Oct 30 17:56:58.069448 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.069394 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 17:56:58.074292 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.074269 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg"] Oct 30 17:56:58.203262 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.203214 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4scwn\" (UniqueName: \"kubernetes.io/projected/244982cc-bbe8-4044-860f-8a766f3562ad-kube-api-access-4scwn\") pod \"mcp-gateway-controller-6c98b995c5-5hfqg\" (UID: \"244982cc-bbe8-4044-860f-8a766f3562ad\") " pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg" Oct 30 17:56:58.305275 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.305060 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4scwn\" (UniqueName: \"kubernetes.io/projected/244982cc-bbe8-4044-860f-8a766f3562ad-kube-api-access-4scwn\") pod \"mcp-gateway-controller-6c98b995c5-5hfqg\" (UID: \"244982cc-bbe8-4044-860f-8a766f3562ad\") " pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg" Oct 30 17:56:58.329866 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.329837 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4scwn\" (UniqueName: \"kubernetes.io/projected/244982cc-bbe8-4044-860f-8a766f3562ad-kube-api-access-4scwn\") pod \"mcp-gateway-controller-6c98b995c5-5hfqg\" (UID: \"244982cc-bbe8-4044-860f-8a766f3562ad\") " pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg" Oct 30 17:56:58.364812 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.364780 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg" Oct 30 17:56:58.370358 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.370316 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9"] Oct 30 17:56:58.374700 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:56:58.374594 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50a26671_57c2_4781_9b78_211ea23c0318.slice/crio-536e6a07ed134f67e03c42f442371973a13db4daa8971f8764202ae7d982e959 WatchSource:0}: Error finding container 536e6a07ed134f67e03c42f442371973a13db4daa8971f8764202ae7d982e959: Status 404 returned error can't find the container with id 536e6a07ed134f67e03c42f442371973a13db4daa8971f8764202ae7d982e959 Oct 30 17:56:58.665278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:58.665248 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg"] Oct 30 17:56:58.668494 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:56:58.668469 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244982cc_bbe8_4044_860f_8a766f3562ad.slice/crio-c2e77030f8b6006cf10b2e7b9dce87b8cf2c28d3e9dc1f7c1fc2d3c246326a05 WatchSource:0}: Error finding container c2e77030f8b6006cf10b2e7b9dce87b8cf2c28d3e9dc1f7c1fc2d3c246326a05: Status 404 returned error can't find the container with id c2e77030f8b6006cf10b2e7b9dce87b8cf2c28d3e9dc1f7c1fc2d3c246326a05 Oct 30 17:56:59.320100 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:59.320055 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg" event={"ID":"244982cc-bbe8-4044-860f-8a766f3562ad","Type":"ContainerStarted","Data":"c2e77030f8b6006cf10b2e7b9dce87b8cf2c28d3e9dc1f7c1fc2d3c246326a05"} Oct 30 17:56:59.321323 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:59.321291 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" event={"ID":"50a26671-57c2-4781-9b78-211ea23c0318","Type":"ContainerStarted","Data":"536e6a07ed134f67e03c42f442371973a13db4daa8971f8764202ae7d982e959"} Oct 30 17:56:59.375446 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:59.375408 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c"] Oct 30 17:56:59.378156 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:59.378132 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c" Oct 30 17:56:59.396629 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:59.396608 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5sbk2\"" Oct 30 17:56:59.410358 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:59.410262 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c"] Oct 30 17:56:59.517210 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:59.517172 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjt7l\" (UniqueName: \"kubernetes.io/projected/6f0a9864-6eec-4854-bcd3-65d07e04367c-kube-api-access-gjt7l\") pod \"limitador-operator-controller-manager-68669df9c4-hb58c\" (UID: \"6f0a9864-6eec-4854-bcd3-65d07e04367c\") " pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c" Oct 30 17:56:59.618736 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:59.618689 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjt7l\" (UniqueName: \"kubernetes.io/projected/6f0a9864-6eec-4854-bcd3-65d07e04367c-kube-api-access-gjt7l\") pod \"limitador-operator-controller-manager-68669df9c4-hb58c\" (UID: \"6f0a9864-6eec-4854-bcd3-65d07e04367c\") " pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c" Oct 30 17:56:59.644000 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:59.643961 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjt7l\" (UniqueName: \"kubernetes.io/projected/6f0a9864-6eec-4854-bcd3-65d07e04367c-kube-api-access-gjt7l\") pod \"limitador-operator-controller-manager-68669df9c4-hb58c\" (UID: \"6f0a9864-6eec-4854-bcd3-65d07e04367c\") " pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c" Oct 30 17:56:59.692554 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:56:59.692522 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c" Oct 30 17:57:00.013312 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:00.013227 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds"] Oct 30 17:57:00.016081 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:00.016051 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds" Oct 30 17:57:00.028495 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:00.028469 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c"] Oct 30 17:57:00.033002 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:57:00.032972 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0a9864_6eec_4854_bcd3_65d07e04367c.slice/crio-de96e9b49cf7f3d725ed475d30b874580364c5138f2b30ef810b8b48fc499054 WatchSource:0}: Error finding container de96e9b49cf7f3d725ed475d30b874580364c5138f2b30ef810b8b48fc499054: Status 404 returned error can't find the container with id de96e9b49cf7f3d725ed475d30b874580364c5138f2b30ef810b8b48fc499054 Oct 30 17:57:00.039693 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:00.039621 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds"] Oct 30 17:57:00.040258 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:00.040222 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"developer-portal-controller-manager-dockercfg-m8bfl\"" Oct 30 17:57:00.123986 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:00.123955 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjmpd\" (UniqueName: \"kubernetes.io/projected/282b0162-4f29-4198-a4dc-50b21b6266b6-kube-api-access-tjmpd\") pod \"developer-portal-controller-66d8dbb8c8-d7cds\" (UID: \"282b0162-4f29-4198-a4dc-50b21b6266b6\") " pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds" Oct 30 17:57:00.225166 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:00.225128 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjmpd\" (UniqueName: \"kubernetes.io/projected/282b0162-4f29-4198-a4dc-50b21b6266b6-kube-api-access-tjmpd\") pod \"developer-portal-controller-66d8dbb8c8-d7cds\" (UID: \"282b0162-4f29-4198-a4dc-50b21b6266b6\") " pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds" Oct 30 17:57:00.251344 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:00.251313 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjmpd\" (UniqueName: \"kubernetes.io/projected/282b0162-4f29-4198-a4dc-50b21b6266b6-kube-api-access-tjmpd\") pod \"developer-portal-controller-66d8dbb8c8-d7cds\" (UID: \"282b0162-4f29-4198-a4dc-50b21b6266b6\") " pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds" Oct 30 17:57:00.327652 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:00.327564 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c" event={"ID":"6f0a9864-6eec-4854-bcd3-65d07e04367c","Type":"ContainerStarted","Data":"de96e9b49cf7f3d725ed475d30b874580364c5138f2b30ef810b8b48fc499054"} Oct 30 17:57:00.335832 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:00.335799 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds" Oct 30 17:57:00.648535 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:00.648500 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds"] Oct 30 17:57:00.658751 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:57:00.658715 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282b0162_4f29_4198_a4dc_50b21b6266b6.slice/crio-f2e07cb62a62cf5eee46884bceaa2fda09686865560eef268943be8f9d2dd39f WatchSource:0}: Error finding container f2e07cb62a62cf5eee46884bceaa2fda09686865560eef268943be8f9d2dd39f: Status 404 returned error can't find the container with id f2e07cb62a62cf5eee46884bceaa2fda09686865560eef268943be8f9d2dd39f Oct 30 17:57:01.335754 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:01.335726 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds" event={"ID":"282b0162-4f29-4198-a4dc-50b21b6266b6","Type":"ContainerStarted","Data":"f2e07cb62a62cf5eee46884bceaa2fda09686865560eef268943be8f9d2dd39f"} Oct 30 17:57:02.342730 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:02.342689 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg" event={"ID":"244982cc-bbe8-4044-860f-8a766f3562ad","Type":"ContainerStarted","Data":"17619b2ccfbc73d782fef4ae04ada742f9917c074398a79f987ad457bb638781"} Oct 30 17:57:02.343232 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:02.342780 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg" Oct 30 17:57:02.344602 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:02.344571 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" event={"ID":"50a26671-57c2-4781-9b78-211ea23c0318","Type":"ContainerStarted","Data":"72df0f26f19edd034b14972c30dccec0d4928dafa13a63329c8855803088f5f5"} Oct 30 17:57:02.344702 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:02.344631 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" Oct 30 17:57:02.393011 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:02.392747 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg" podStartSLOduration=1.404226296 podStartE2EDuration="4.392729283s" totalImagesPullingTime="2.988502987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:02.392422092 +0000 UTC m=+281.349425930" watchObservedRunningTime="2026-10-30 17:57:02.392729283 +0000 UTC m=+281.349733120" Oct 30 17:57:02.448195 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:02.448133 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podStartSLOduration=2.497061641 podStartE2EDuration="5.448115256s" totalImagesPullingTime="2.951053615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:02.447371104 +0000 UTC m=+281.404374940" watchObservedRunningTime="2026-10-30 17:57:02.448115256 +0000 UTC m=+281.405119094" Oct 30 17:57:03.351257 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.351218 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c" event={"ID":"6f0a9864-6eec-4854-bcd3-65d07e04367c","Type":"ContainerStarted","Data":"7cf50e407405a158b10fb2c531627629cd1b6b7b164a78a2693d707c054922b3"} Oct 30 17:57:03.351732 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.351415 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c" Oct 30 17:57:03.396286 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.396221 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c" podStartSLOduration=1.557694606 podStartE2EDuration="4.396200055s" totalImagesPullingTime="2.838505449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:03.395095291 +0000 UTC m=+282.352099127" watchObservedRunningTime="2026-10-30 17:57:03.396200055 +0000 UTC m=+282.353203893" Oct 30 17:57:03.722474 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.722396 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66"] Oct 30 17:57:03.724709 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.724689 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" Oct 30 17:57:03.750476 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.750445 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66"] Oct 30 17:57:03.760372 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.760335 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 17:57:03.760716 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.760682 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 17:57:03.761460 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.760719 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bn27s\"" Oct 30 17:57:03.863272 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.863232 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e6711b6-3b88-438e-958e-f64432be2ddc-tls-certs\") pod \"kube-auth-proxy-9c55b8cc4-f2d66\" (UID: \"5e6711b6-3b88-438e-958e-f64432be2ddc\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" Oct 30 17:57:03.863425 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.863317 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e6711b6-3b88-438e-958e-f64432be2ddc-tmp\") pod \"kube-auth-proxy-9c55b8cc4-f2d66\" (UID: \"5e6711b6-3b88-438e-958e-f64432be2ddc\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" Oct 30 17:57:03.863425 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.863356 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2d7\" (UniqueName: \"kubernetes.io/projected/5e6711b6-3b88-438e-958e-f64432be2ddc-kube-api-access-bz2d7\") pod \"kube-auth-proxy-9c55b8cc4-f2d66\" (UID: \"5e6711b6-3b88-438e-958e-f64432be2ddc\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" Oct 30 17:57:03.964751 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.964709 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2d7\" (UniqueName: \"kubernetes.io/projected/5e6711b6-3b88-438e-958e-f64432be2ddc-kube-api-access-bz2d7\") pod \"kube-auth-proxy-9c55b8cc4-f2d66\" (UID: \"5e6711b6-3b88-438e-958e-f64432be2ddc\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" Oct 30 17:57:03.964957 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.964856 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e6711b6-3b88-438e-958e-f64432be2ddc-tls-certs\") pod \"kube-auth-proxy-9c55b8cc4-f2d66\" (UID: \"5e6711b6-3b88-438e-958e-f64432be2ddc\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" Oct 30 17:57:03.964957 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.964881 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e6711b6-3b88-438e-958e-f64432be2ddc-tmp\") pod \"kube-auth-proxy-9c55b8cc4-f2d66\" (UID: \"5e6711b6-3b88-438e-958e-f64432be2ddc\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" Oct 30 17:57:03.967245 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.967219 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e6711b6-3b88-438e-958e-f64432be2ddc-tmp\") pod \"kube-auth-proxy-9c55b8cc4-f2d66\" (UID: \"5e6711b6-3b88-438e-958e-f64432be2ddc\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" Oct 30 17:57:03.967563 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.967543 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e6711b6-3b88-438e-958e-f64432be2ddc-tls-certs\") pod \"kube-auth-proxy-9c55b8cc4-f2d66\" (UID: \"5e6711b6-3b88-438e-958e-f64432be2ddc\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" Oct 30 17:57:03.993481 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:03.993415 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2d7\" (UniqueName: \"kubernetes.io/projected/5e6711b6-3b88-438e-958e-f64432be2ddc-kube-api-access-bz2d7\") pod \"kube-auth-proxy-9c55b8cc4-f2d66\" (UID: \"5e6711b6-3b88-438e-958e-f64432be2ddc\") " pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" Oct 30 17:57:04.036673 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:04.036623 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" Oct 30 17:57:04.345612 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:04.345580 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66"] Oct 30 17:57:04.348893 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:57:04.348859 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6711b6_3b88_438e_958e_f64432be2ddc.slice/crio-dd50c8fe168aa566d6c9cac4c64a74800bf5cd1649d90e9a92af9a080e1027b2 WatchSource:0}: Error finding container dd50c8fe168aa566d6c9cac4c64a74800bf5cd1649d90e9a92af9a080e1027b2: Status 404 returned error can't find the container with id dd50c8fe168aa566d6c9cac4c64a74800bf5cd1649d90e9a92af9a080e1027b2 Oct 30 17:57:04.358001 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:04.357972 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" event={"ID":"5e6711b6-3b88-438e-958e-f64432be2ddc","Type":"ContainerStarted","Data":"dd50c8fe168aa566d6c9cac4c64a74800bf5cd1649d90e9a92af9a080e1027b2"} Oct 30 17:57:05.089257 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:57:05.089215 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:57:05.391316 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.391276 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-8bwz6"] Oct 30 17:57:05.397169 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.397141 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" Oct 30 17:57:05.414161 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.414132 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 17:57:05.417098 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.417067 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-8bwz6"] Oct 30 17:57:05.428519 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.428489 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xzxsc\"" Oct 30 17:57:05.487265 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.487233 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/adc8b3e9-ed00-48a6-8688-379f585e65b4-config-file\") pod \"limitador-limitador-77dd99c685-8bwz6\" (UID: \"adc8b3e9-ed00-48a6-8688-379f585e65b4\") " pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" Oct 30 17:57:05.487442 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.487326 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbstn\" (UniqueName: \"kubernetes.io/projected/adc8b3e9-ed00-48a6-8688-379f585e65b4-kube-api-access-hbstn\") pod \"limitador-limitador-77dd99c685-8bwz6\" (UID: \"adc8b3e9-ed00-48a6-8688-379f585e65b4\") " pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" Oct 30 17:57:05.533439 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.533402 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-8bwz6"] Oct 30 17:57:05.588238 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.588195 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/adc8b3e9-ed00-48a6-8688-379f585e65b4-config-file\") pod \"limitador-limitador-77dd99c685-8bwz6\" (UID: \"adc8b3e9-ed00-48a6-8688-379f585e65b4\") " pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" Oct 30 17:57:05.588425 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.588319 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbstn\" (UniqueName: \"kubernetes.io/projected/adc8b3e9-ed00-48a6-8688-379f585e65b4-kube-api-access-hbstn\") pod \"limitador-limitador-77dd99c685-8bwz6\" (UID: \"adc8b3e9-ed00-48a6-8688-379f585e65b4\") " pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" Oct 30 17:57:05.588973 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.588950 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/adc8b3e9-ed00-48a6-8688-379f585e65b4-config-file\") pod \"limitador-limitador-77dd99c685-8bwz6\" (UID: \"adc8b3e9-ed00-48a6-8688-379f585e65b4\") " pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" Oct 30 17:57:05.613461 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.613430 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbstn\" (UniqueName: \"kubernetes.io/projected/adc8b3e9-ed00-48a6-8688-379f585e65b4-kube-api-access-hbstn\") pod \"limitador-limitador-77dd99c685-8bwz6\" (UID: \"adc8b3e9-ed00-48a6-8688-379f585e65b4\") " pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" Oct 30 17:57:05.712455 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:05.712364 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" Oct 30 17:57:06.027003 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:06.026885 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-8bwz6"] Oct 30 17:57:06.032146 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:57:06.032107 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc8b3e9_ed00_48a6_8688_379f585e65b4.slice/crio-570f62068919cc429eb24fde969b0f3d1a15ce3b429cbcc3156e56b9f81784c3 WatchSource:0}: Error finding container 570f62068919cc429eb24fde969b0f3d1a15ce3b429cbcc3156e56b9f81784c3: Status 404 returned error can't find the container with id 570f62068919cc429eb24fde969b0f3d1a15ce3b429cbcc3156e56b9f81784c3 Oct 30 17:57:06.369998 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:06.369963 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" event={"ID":"adc8b3e9-ed00-48a6-8688-379f585e65b4","Type":"ContainerStarted","Data":"570f62068919cc429eb24fde969b0f3d1a15ce3b429cbcc3156e56b9f81784c3"} Oct 30 17:57:07.376293 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:07.376244 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" event={"ID":"adc8b3e9-ed00-48a6-8688-379f585e65b4","Type":"ContainerStarted","Data":"b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943"} Oct 30 17:57:07.376814 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:07.376328 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" Oct 30 17:57:07.377879 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:07.377855 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" event={"ID":"5e6711b6-3b88-438e-958e-f64432be2ddc","Type":"ContainerStarted","Data":"d124f750859d9a2fdaf6df6a322fd7c9da8eb328fa6516b20bd04ad172f75ef3"} Oct 30 17:57:07.419695 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:07.419572 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" podStartSLOduration=1.307947369 podStartE2EDuration="2.419553311s" totalImagesPullingTime="1.111605942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:07.41941159 +0000 UTC m=+286.376415427" watchObservedRunningTime="2026-10-30 17:57:07.419553311 +0000 UTC m=+286.376557156" Oct 30 17:57:07.463596 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:07.463538 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9c55b8cc4-f2d66" podStartSLOduration=2.040441142 podStartE2EDuration="4.463523263s" totalImagesPullingTime="2.423082121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:07.462157115 +0000 UTC m=+286.419160963" watchObservedRunningTime="2026-10-30 17:57:07.463523263 +0000 UTC m=+286.420527096" Oct 30 17:57:07.748227 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:07.748152 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-dc8775675-n5vjm"] Oct 30 17:57:07.750498 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:07.750481 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc8775675-n5vjm" Oct 30 17:57:07.778629 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:07.778594 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dc8775675-n5vjm"] Oct 30 17:57:07.832994 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:07.832964 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fnvtk\"" Oct 30 17:57:07.909471 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:07.909437 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v56g5\" (UniqueName: \"kubernetes.io/projected/d4b3ed18-17ff-44c8-b38d-c67730d0f6c6-kube-api-access-v56g5\") pod \"authorino-dc8775675-n5vjm\" (UID: \"d4b3ed18-17ff-44c8-b38d-c67730d0f6c6\") " pod="kuadrant-system/authorino-dc8775675-n5vjm" Oct 30 17:57:08.009920 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:08.009831 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v56g5\" (UniqueName: \"kubernetes.io/projected/d4b3ed18-17ff-44c8-b38d-c67730d0f6c6-kube-api-access-v56g5\") pod \"authorino-dc8775675-n5vjm\" (UID: \"d4b3ed18-17ff-44c8-b38d-c67730d0f6c6\") " pod="kuadrant-system/authorino-dc8775675-n5vjm" Oct 30 17:57:08.034471 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:08.034435 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v56g5\" (UniqueName: \"kubernetes.io/projected/d4b3ed18-17ff-44c8-b38d-c67730d0f6c6-kube-api-access-v56g5\") pod \"authorino-dc8775675-n5vjm\" (UID: \"d4b3ed18-17ff-44c8-b38d-c67730d0f6c6\") " pod="kuadrant-system/authorino-dc8775675-n5vjm" Oct 30 17:57:08.061209 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:08.061181 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc8775675-n5vjm" Oct 30 17:57:08.354385 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:08.354355 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" Oct 30 17:57:08.363809 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:08.363783 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dc8775675-n5vjm"] Oct 30 17:57:08.368117 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:57:08.368088 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4b3ed18_17ff_44c8_b38d_c67730d0f6c6.slice/crio-c653070b31b90009bdae64c9b279f651a55329b05eaeb9ab58738a292a6a01aa WatchSource:0}: Error finding container c653070b31b90009bdae64c9b279f651a55329b05eaeb9ab58738a292a6a01aa: Status 404 returned error can't find the container with id c653070b31b90009bdae64c9b279f651a55329b05eaeb9ab58738a292a6a01aa Oct 30 17:57:08.383278 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:08.383252 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc8775675-n5vjm" event={"ID":"d4b3ed18-17ff-44c8-b38d-c67730d0f6c6","Type":"ContainerStarted","Data":"c653070b31b90009bdae64c9b279f651a55329b05eaeb9ab58738a292a6a01aa"} Oct 30 17:57:10.395386 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:10.395348 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds" event={"ID":"282b0162-4f29-4198-a4dc-50b21b6266b6","Type":"ContainerStarted","Data":"8e03166b4b30e4330a1616992cdbb720e5bf0816cd525b7d6c51a899aa73937a"} Oct 30 17:57:10.395872 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:10.395437 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds" Oct 30 17:57:10.396913 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:10.396890 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc8775675-n5vjm" event={"ID":"d4b3ed18-17ff-44c8-b38d-c67730d0f6c6","Type":"ContainerStarted","Data":"9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f"} Oct 30 17:57:10.440026 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:10.439974 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds" podStartSLOduration=2.219989245 podStartE2EDuration="11.439962493s" totalImagesPullingTime="9.219973248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:10.439454618 +0000 UTC m=+289.396458455" watchObservedRunningTime="2026-10-30 17:57:10.439962493 +0000 UTC m=+289.396966328" Oct 30 17:57:10.502406 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:10.502351 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-dc8775675-n5vjm" podStartSLOduration=1.8881718140000001 podStartE2EDuration="3.502336686s" totalImagesPullingTime="1.614164872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:10.501361548 +0000 UTC m=+289.458365384" watchObservedRunningTime="2026-10-30 17:57:10.502336686 +0000 UTC m=+289.459340522" Oct 30 17:57:13.353612 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:13.353579 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/mcp-gateway-controller-6c98b995c5-5hfqg" Oct 30 17:57:14.360951 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:14.360907 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-hb58c" Oct 30 17:57:18.385077 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:18.385037 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" Oct 30 17:57:21.201519 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:21.201495 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 17:57:21.201953 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:21.201525 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 17:57:21.213075 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:21.213050 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 17:57:21.213311 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:21.213297 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 17:57:21.402832 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:21.402803 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-d7cds" Oct 30 17:57:22.606082 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:22.606038 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-8bwz6"] Oct 30 17:57:22.606549 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:22.606287 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" podUID="adc8b3e9-ed00-48a6-8688-379f585e65b4" containerName="limitador" containerID="cri-o://b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943" gracePeriod=30 Oct 30 17:57:23.151240 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.151217 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" Oct 30 17:57:23.223178 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.223093 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc8b3e9-ed00-48a6-8688-379f585e65b4-config-file\" (UniqueName: \"kubernetes.io/configmap/adc8b3e9-ed00-48a6-8688-379f585e65b4-config-file\") pod \"adc8b3e9-ed00-48a6-8688-379f585e65b4\" (UID: \"adc8b3e9-ed00-48a6-8688-379f585e65b4\") " Oct 30 17:57:23.223348 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.223221 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc8b3e9-ed00-48a6-8688-379f585e65b4-kube-api-access-hbstn\" (UniqueName: \"kubernetes.io/projected/adc8b3e9-ed00-48a6-8688-379f585e65b4-kube-api-access-hbstn\") pod \"adc8b3e9-ed00-48a6-8688-379f585e65b4\" (UID: \"adc8b3e9-ed00-48a6-8688-379f585e65b4\") " Oct 30 17:57:23.223473 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.223446 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc8b3e9-ed00-48a6-8688-379f585e65b4-config-file" pod "adc8b3e9-ed00-48a6-8688-379f585e65b4" (UID: "adc8b3e9-ed00-48a6-8688-379f585e65b4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:23.225441 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.225421 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc8b3e9-ed00-48a6-8688-379f585e65b4-kube-api-access-hbstn" pod "adc8b3e9-ed00-48a6-8688-379f585e65b4" (UID: "adc8b3e9-ed00-48a6-8688-379f585e65b4"). InnerVolumeSpecName "kube-api-access-hbstn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:23.323839 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.323797 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbstn\" (UniqueName: \"kubernetes.io/projected/adc8b3e9-ed00-48a6-8688-379f585e65b4-kube-api-access-hbstn\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:57:23.323839 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.323832 4883 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/adc8b3e9-ed00-48a6-8688-379f585e65b4-config-file\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:57:23.453155 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.453119 4883 generic.go:356] "Generic (PLEG): container finished" podID="adc8b3e9-ed00-48a6-8688-379f585e65b4" containerID="b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943" exitCode=0 Oct 30 17:57:23.453303 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.453169 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" Oct 30 17:57:23.453303 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.453186 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" event={"ID":"adc8b3e9-ed00-48a6-8688-379f585e65b4","Type":"ContainerDied","Data":"b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943"} Oct 30 17:57:23.453303 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.453228 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-77dd99c685-8bwz6" event={"ID":"adc8b3e9-ed00-48a6-8688-379f585e65b4","Type":"ContainerDied","Data":"570f62068919cc429eb24fde969b0f3d1a15ce3b429cbcc3156e56b9f81784c3"} Oct 30 17:57:23.453303 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.453248 4883 scope.go:122] "RemoveContainer" containerID="b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943" Oct 30 17:57:23.461846 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.461827 4883 scope.go:122] "RemoveContainer" containerID="b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943" Oct 30 17:57:23.462144 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:57:23.462125 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943\": container with ID starting with b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943 not found: ID does not exist" containerID="b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943" Oct 30 17:57:23.462220 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.462157 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943"} err="failed to get container status \"b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943\": rpc error: code = NotFound desc = could not find container \"b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943\": container with ID starting with b6b19dc590c1624d1a3da7b422abe3bddda94c69397883558d506ce223202943 not found: ID does not exist" Oct 30 17:57:23.510213 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.510139 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-8bwz6"] Oct 30 17:57:23.514041 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:23.514021 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-77dd99c685-8bwz6"] Oct 30 17:57:25.179550 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:25.179512 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc8b3e9-ed00-48a6-8688-379f585e65b4" path="/var/lib/kubelet/pods/adc8b3e9-ed00-48a6-8688-379f585e65b4/volumes" Oct 30 17:57:35.077234 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:57:35.077153 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:57:55.391021 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.390982 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8689bcdb59-997v7"] Oct 30 17:57:55.391568 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.391550 4883 cpu_manager.go:414] "containerMap: removing container" podUID="adc8b3e9-ed00-48a6-8688-379f585e65b4" containerName="limitador" Oct 30 17:57:55.391651 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.391570 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc8b3e9-ed00-48a6-8688-379f585e65b4" containerName="limitador" Oct 30 17:57:55.391651 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.391625 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc8b3e9-ed00-48a6-8688-379f585e65b4" containerName="limitador" Oct 30 17:57:55.402121 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.402099 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:57:55.418366 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.418341 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8689bcdb59-997v7"] Oct 30 17:57:55.429883 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.429852 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 17:57:55.430064 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.430048 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-whrck\"" Oct 30 17:57:55.430130 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.430117 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 17:57:55.430175 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.430136 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 17:57:55.431960 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.431928 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 17:57:55.432036 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.431965 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 17:57:55.485817 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.485789 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-metrics-certs\") pod \"maas-controller-8689bcdb59-997v7\" (UID: \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\") " pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:57:55.485983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.485857 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh9p8\" (UniqueName: \"kubernetes.io/projected/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-kube-api-access-jh9p8\") pod \"maas-controller-8689bcdb59-997v7\" (UID: \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\") " pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:57:55.485983 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.485961 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-webhook-cert\") pod \"maas-controller-8689bcdb59-997v7\" (UID: \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\") " pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:57:55.587071 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.587031 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh9p8\" (UniqueName: \"kubernetes.io/projected/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-kube-api-access-jh9p8\") pod \"maas-controller-8689bcdb59-997v7\" (UID: \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\") " pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:57:55.587241 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.587080 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-webhook-cert\") pod \"maas-controller-8689bcdb59-997v7\" (UID: \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\") " pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:57:55.587241 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.587120 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-metrics-certs\") pod \"maas-controller-8689bcdb59-997v7\" (UID: \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\") " pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:57:55.589678 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.589651 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-metrics-certs\") pod \"maas-controller-8689bcdb59-997v7\" (UID: \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\") " pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:57:55.589798 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.589676 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-webhook-cert\") pod \"maas-controller-8689bcdb59-997v7\" (UID: \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\") " pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:57:55.612325 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.612298 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh9p8\" (UniqueName: \"kubernetes.io/projected/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-kube-api-access-jh9p8\") pod \"maas-controller-8689bcdb59-997v7\" (UID: \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\") " pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:57:55.714756 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.714666 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:57:55.908418 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.908384 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5456657644-j2hgt"] Oct 30 17:57:55.912813 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.912795 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:57:55.938128 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.938099 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5456657644-j2hgt"] Oct 30 17:57:55.990524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.990450 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/599aec62-8914-4abe-a7e0-3c70148780e6-webhook-cert\") pod \"maas-controller-5456657644-j2hgt\" (UID: \"599aec62-8914-4abe-a7e0-3c70148780e6\") " pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:57:55.990524 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.990495 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svz59\" (UniqueName: \"kubernetes.io/projected/599aec62-8914-4abe-a7e0-3c70148780e6-kube-api-access-svz59\") pod \"maas-controller-5456657644-j2hgt\" (UID: \"599aec62-8914-4abe-a7e0-3c70148780e6\") " pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:57:55.990713 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:55.990617 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599aec62-8914-4abe-a7e0-3c70148780e6-metrics-certs\") pod \"maas-controller-5456657644-j2hgt\" (UID: \"599aec62-8914-4abe-a7e0-3c70148780e6\") " pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:57:56.020523 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:56.020490 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8689bcdb59-997v7"] Oct 30 17:57:56.024339 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:57:56.024307 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8b52b1_5b81_4db9_93aa_0e1f32eb2ced.slice/crio-146ba3aeeefe36fa17eabc074a5d48b57fe2970a24a34d9458ea6167de5d3c4e WatchSource:0}: Error finding container 146ba3aeeefe36fa17eabc074a5d48b57fe2970a24a34d9458ea6167de5d3c4e: Status 404 returned error can't find the container with id 146ba3aeeefe36fa17eabc074a5d48b57fe2970a24a34d9458ea6167de5d3c4e Oct 30 17:57:56.025617 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:56.025599 4883 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:57:56.091778 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:56.091743 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599aec62-8914-4abe-a7e0-3c70148780e6-metrics-certs\") pod \"maas-controller-5456657644-j2hgt\" (UID: \"599aec62-8914-4abe-a7e0-3c70148780e6\") " pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:57:56.091914 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:56.091790 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/599aec62-8914-4abe-a7e0-3c70148780e6-webhook-cert\") pod \"maas-controller-5456657644-j2hgt\" (UID: \"599aec62-8914-4abe-a7e0-3c70148780e6\") " pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:57:56.091914 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:56.091823 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svz59\" (UniqueName: \"kubernetes.io/projected/599aec62-8914-4abe-a7e0-3c70148780e6-kube-api-access-svz59\") pod \"maas-controller-5456657644-j2hgt\" (UID: \"599aec62-8914-4abe-a7e0-3c70148780e6\") " pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:57:56.094381 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:56.094354 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/599aec62-8914-4abe-a7e0-3c70148780e6-webhook-cert\") pod \"maas-controller-5456657644-j2hgt\" (UID: \"599aec62-8914-4abe-a7e0-3c70148780e6\") " pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:57:56.094487 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:56.094354 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599aec62-8914-4abe-a7e0-3c70148780e6-metrics-certs\") pod \"maas-controller-5456657644-j2hgt\" (UID: \"599aec62-8914-4abe-a7e0-3c70148780e6\") " pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:57:56.125001 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:56.124980 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svz59\" (UniqueName: \"kubernetes.io/projected/599aec62-8914-4abe-a7e0-3c70148780e6-kube-api-access-svz59\") pod \"maas-controller-5456657644-j2hgt\" (UID: \"599aec62-8914-4abe-a7e0-3c70148780e6\") " pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:57:56.225241 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:56.225202 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:57:56.526238 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:56.526154 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5456657644-j2hgt"] Oct 30 17:57:56.529493 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:57:56.529464 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599aec62_8914_4abe_a7e0_3c70148780e6.slice/crio-df48abd54194be9a0087d664b964c6675825a2aa24f85905f071b558bd564c11 WatchSource:0}: Error finding container df48abd54194be9a0087d664b964c6675825a2aa24f85905f071b558bd564c11: Status 404 returned error can't find the container with id df48abd54194be9a0087d664b964c6675825a2aa24f85905f071b558bd564c11 Oct 30 17:57:56.603995 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:56.603960 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689bcdb59-997v7" event={"ID":"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced","Type":"ContainerStarted","Data":"146ba3aeeefe36fa17eabc074a5d48b57fe2970a24a34d9458ea6167de5d3c4e"} Oct 30 17:57:56.605212 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:56.605187 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5456657644-j2hgt" event={"ID":"599aec62-8914-4abe-a7e0-3c70148780e6","Type":"ContainerStarted","Data":"df48abd54194be9a0087d664b964c6675825a2aa24f85905f071b558bd564c11"} Oct 30 17:57:59.625186 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:59.625151 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689bcdb59-997v7" event={"ID":"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced","Type":"ContainerStarted","Data":"4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01"} Oct 30 17:57:59.625677 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:59.625275 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:57:59.626701 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:59.626675 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5456657644-j2hgt" event={"ID":"599aec62-8914-4abe-a7e0-3c70148780e6","Type":"ContainerStarted","Data":"b5e32539aeec6a798df4d5e1e987f550fa0bb88a59233fa616c00627eebc53be"} Oct 30 17:57:59.626799 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:59.626732 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:57:59.720283 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:59.720218 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8689bcdb59-997v7" podStartSLOduration=1.747459439 podStartE2EDuration="4.720203316s" totalImagesPullingTime="2.972743877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:59.718978891 +0000 UTC m=+338.675982728" watchObservedRunningTime="2026-10-30 17:57:59.720203316 +0000 UTC m=+338.677207152" Oct 30 17:57:59.773111 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:57:59.773056 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5456657644-j2hgt" podStartSLOduration=2.056816247 podStartE2EDuration="4.773033958s" totalImagesPullingTime="2.716217711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:59.771543809 +0000 UTC m=+338.728547645" watchObservedRunningTime="2026-10-30 17:57:59.773033958 +0000 UTC m=+338.730037794" Oct 30 17:58:01.653763 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.653726 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f"] Oct 30 17:58:01.657489 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.657470 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:01.679967 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.679902 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f"] Oct 30 17:58:01.691349 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.691318 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 17:58:01.691549 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.691343 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-gtw4g\"" Oct 30 17:58:01.691712 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.691408 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 17:58:01.691712 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.691425 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 17:58:01.752871 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.752837 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:01.753093 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.752894 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldv4\" (UniqueName: \"kubernetes.io/projected/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-kube-api-access-nldv4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:01.753093 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.753054 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:01.753208 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.753141 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:01.854536 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.854494 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldv4\" (UniqueName: \"kubernetes.io/projected/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-kube-api-access-nldv4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:01.854751 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.854546 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:01.854751 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.854582 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:01.854751 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.854630 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:01.854751 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:58:01.854745 4883 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 17:58:01.854965 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:58:01.854813 4883 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-tls-cert podName:eee8dbb1-8a5b-4600-893d-78ecd9b403fe nodeName:}" failed. No retries permitted until 2026-10-30 17:58:02.354792578 +0000 UTC m=+341.311796406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" (UID: "eee8dbb1-8a5b-4600-893d-78ecd9b403fe") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 17:58:01.855338 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.855315 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:01.855529 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.855508 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:01.880200 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:01.880166 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldv4\" (UniqueName: \"kubernetes.io/projected/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-kube-api-access-nldv4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:02.360037 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:02.359995 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:02.362388 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:02.362367 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eee8dbb1-8a5b-4600-893d-78ecd9b403fe-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xz9f\" (UID: \"eee8dbb1-8a5b-4600-893d-78ecd9b403fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:02.571248 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:02.571208 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:02.888046 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:02.887962 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f"] Oct 30 17:58:02.891087 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:58:02.891055 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee8dbb1_8a5b_4600_893d_78ecd9b403fe.slice/crio-df81982f258db06c262bbf07d9abf99cb9ff61c546058d3e3287ebca466899fe WatchSource:0}: Error finding container df81982f258db06c262bbf07d9abf99cb9ff61c546058d3e3287ebca466899fe: Status 404 returned error can't find the container with id df81982f258db06c262bbf07d9abf99cb9ff61c546058d3e3287ebca466899fe Oct 30 17:58:03.655836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:03.655791 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" event={"ID":"eee8dbb1-8a5b-4600-893d-78ecd9b403fe","Type":"ContainerStarted","Data":"df81982f258db06c262bbf07d9abf99cb9ff61c546058d3e3287ebca466899fe"} Oct 30 17:58:05.108597 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:58:05.108531 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:58:05.732429 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.732390 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp"] Oct 30 17:58:05.737664 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.737632 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" Oct 30 17:58:05.763364 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.763283 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp"] Oct 30 17:58:05.771784 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.771652 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jd8ds\"" Oct 30 17:58:05.771784 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.771743 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 17:58:05.819420 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.819383 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-d66qd"] Oct 30 17:58:05.823254 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.823231 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:05.844762 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.844729 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-d66qd"] Oct 30 17:58:05.895227 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.895188 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/022f46a3-733f-49cc-8190-466a2d40d766-metrics-certs\") pod \"odh-model-controller-769875db56-d66qd\" (UID: \"022f46a3-733f-49cc-8190-466a2d40d766\") " pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:05.895419 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.895317 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/764b4161-26c5-4cba-a529-619ccef2bdab-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-gtngp\" (UID: \"764b4161-26c5-4cba-a529-619ccef2bdab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" Oct 30 17:58:05.895419 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.895378 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvm7k\" (UniqueName: \"kubernetes.io/projected/022f46a3-733f-49cc-8190-466a2d40d766-kube-api-access-bvm7k\") pod \"odh-model-controller-769875db56-d66qd\" (UID: \"022f46a3-733f-49cc-8190-466a2d40d766\") " pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:05.895419 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.895413 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/022f46a3-733f-49cc-8190-466a2d40d766-cert\") pod \"odh-model-controller-769875db56-d66qd\" (UID: \"022f46a3-733f-49cc-8190-466a2d40d766\") " pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:05.895640 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.895462 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghr88\" (UniqueName: \"kubernetes.io/projected/764b4161-26c5-4cba-a529-619ccef2bdab-kube-api-access-ghr88\") pod \"llmisvc-controller-manager-869d5dcc5d-gtngp\" (UID: \"764b4161-26c5-4cba-a529-619ccef2bdab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" Oct 30 17:58:05.915381 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.915343 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 17:58:05.915547 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.915417 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ls5tc\"" Oct 30 17:58:05.915547 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.915451 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 17:58:05.996486 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.996406 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/764b4161-26c5-4cba-a529-619ccef2bdab-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-gtngp\" (UID: \"764b4161-26c5-4cba-a529-619ccef2bdab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" Oct 30 17:58:05.996746 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.996483 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvm7k\" (UniqueName: \"kubernetes.io/projected/022f46a3-733f-49cc-8190-466a2d40d766-kube-api-access-bvm7k\") pod \"odh-model-controller-769875db56-d66qd\" (UID: \"022f46a3-733f-49cc-8190-466a2d40d766\") " pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:05.996746 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.996524 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/022f46a3-733f-49cc-8190-466a2d40d766-cert\") pod \"odh-model-controller-769875db56-d66qd\" (UID: \"022f46a3-733f-49cc-8190-466a2d40d766\") " pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:05.996746 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.996569 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghr88\" (UniqueName: \"kubernetes.io/projected/764b4161-26c5-4cba-a529-619ccef2bdab-kube-api-access-ghr88\") pod \"llmisvc-controller-manager-869d5dcc5d-gtngp\" (UID: \"764b4161-26c5-4cba-a529-619ccef2bdab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" Oct 30 17:58:05.996746 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.996637 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/022f46a3-733f-49cc-8190-466a2d40d766-metrics-certs\") pod \"odh-model-controller-769875db56-d66qd\" (UID: \"022f46a3-733f-49cc-8190-466a2d40d766\") " pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:05.999654 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:05.999627 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/022f46a3-733f-49cc-8190-466a2d40d766-metrics-certs\") pod \"odh-model-controller-769875db56-d66qd\" (UID: \"022f46a3-733f-49cc-8190-466a2d40d766\") " pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:06.001414 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:06.001392 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/022f46a3-733f-49cc-8190-466a2d40d766-cert\") pod \"odh-model-controller-769875db56-d66qd\" (UID: \"022f46a3-733f-49cc-8190-466a2d40d766\") " pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:06.002428 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:06.002407 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/764b4161-26c5-4cba-a529-619ccef2bdab-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-gtngp\" (UID: \"764b4161-26c5-4cba-a529-619ccef2bdab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" Oct 30 17:58:06.023453 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:06.023420 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghr88\" (UniqueName: \"kubernetes.io/projected/764b4161-26c5-4cba-a529-619ccef2bdab-kube-api-access-ghr88\") pod \"llmisvc-controller-manager-869d5dcc5d-gtngp\" (UID: \"764b4161-26c5-4cba-a529-619ccef2bdab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" Oct 30 17:58:06.028323 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:06.028294 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvm7k\" (UniqueName: \"kubernetes.io/projected/022f46a3-733f-49cc-8190-466a2d40d766-kube-api-access-bvm7k\") pod \"odh-model-controller-769875db56-d66qd\" (UID: \"022f46a3-733f-49cc-8190-466a2d40d766\") " pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:06.053198 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:06.052994 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" Oct 30 17:58:06.139332 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:06.139128 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:06.372365 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:06.372317 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp"] Oct 30 17:58:06.374873 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:58:06.374834 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod764b4161_26c5_4cba_a529_619ccef2bdab.slice/crio-bafdab0b10a848348121e26170aea83ac4a426f570203acc7a10d1060433df0b WatchSource:0}: Error finding container bafdab0b10a848348121e26170aea83ac4a426f570203acc7a10d1060433df0b: Status 404 returned error can't find the container with id bafdab0b10a848348121e26170aea83ac4a426f570203acc7a10d1060433df0b Oct 30 17:58:06.455840 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:06.455802 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-d66qd"] Oct 30 17:58:06.460576 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:58:06.460547 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022f46a3_733f_49cc_8190_466a2d40d766.slice/crio-10b45a340ac6211f4d830969aa7dbdfe1ffd68020b5940aad893a3f5e173aac7 WatchSource:0}: Error finding container 10b45a340ac6211f4d830969aa7dbdfe1ffd68020b5940aad893a3f5e173aac7: Status 404 returned error can't find the container with id 10b45a340ac6211f4d830969aa7dbdfe1ffd68020b5940aad893a3f5e173aac7 Oct 30 17:58:06.679082 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:06.678983 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" event={"ID":"764b4161-26c5-4cba-a529-619ccef2bdab","Type":"ContainerStarted","Data":"bafdab0b10a848348121e26170aea83ac4a426f570203acc7a10d1060433df0b"} Oct 30 17:58:06.680464 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:06.680433 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-d66qd" event={"ID":"022f46a3-733f-49cc-8190-466a2d40d766","Type":"ContainerStarted","Data":"10b45a340ac6211f4d830969aa7dbdfe1ffd68020b5940aad893a3f5e173aac7"} Oct 30 17:58:09.701355 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:09.701309 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" event={"ID":"764b4161-26c5-4cba-a529-619ccef2bdab","Type":"ContainerStarted","Data":"9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2"} Oct 30 17:58:09.701836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:09.701481 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" Oct 30 17:58:09.746745 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:09.746685 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" podStartSLOduration=1.8444789670000001 podStartE2EDuration="4.746668999s" totalImagesPullingTime="2.902190032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:09.745252151 +0000 UTC m=+348.702255988" watchObservedRunningTime="2026-10-30 17:58:09.746668999 +0000 UTC m=+348.703672862" Oct 30 17:58:10.633423 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:10.633385 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5456657644-j2hgt" Oct 30 17:58:10.633629 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:10.633608 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:58:10.708825 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:10.708775 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-d66qd" event={"ID":"022f46a3-733f-49cc-8190-466a2d40d766","Type":"ContainerStarted","Data":"8b9b0cd943d41960c7e3ed06b6fbb0c13d5d11ec8e707dce01df90e1af1f20c0"} Oct 30 17:58:10.709294 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:10.708890 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:10.850831 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:10.850780 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-d66qd" podStartSLOduration=2.411139313 podStartE2EDuration="5.8507642s" totalImagesPullingTime="3.439624887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:10.847988848 +0000 UTC m=+349.804992686" watchObservedRunningTime="2026-10-30 17:58:10.8507642 +0000 UTC m=+349.807768037" Oct 30 17:58:10.898541 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:10.898447 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8689bcdb59-997v7"] Oct 30 17:58:10.898716 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:10.898674 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8689bcdb59-997v7" podUID="3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced" containerName="manager" containerID="cri-o://4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01" gracePeriod=30 Oct 30 17:58:11.159155 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.159086 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:58:11.248986 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.248924 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-kube-api-access-jh9p8\" (UniqueName: \"kubernetes.io/projected/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-kube-api-access-jh9p8\") pod \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\" (UID: \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\") " Oct 30 17:58:11.249155 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.249006 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-metrics-certs\") pod \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\" (UID: \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\") " Oct 30 17:58:11.249155 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.249084 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-webhook-cert\") pod \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\" (UID: \"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced\") " Oct 30 17:58:11.251642 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.251605 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-kube-api-access-jh9p8" pod "3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced" (UID: "3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced"). InnerVolumeSpecName "kube-api-access-jh9p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:11.251759 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.251603 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-metrics-certs" pod "3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced" (UID: "3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:11.251866 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.251840 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-webhook-cert" pod "3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced" (UID: "3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:11.350576 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.350540 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh9p8\" (UniqueName: \"kubernetes.io/projected/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-kube-api-access-jh9p8\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:58:11.350576 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.350581 4883 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-metrics-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:58:11.350801 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.350593 4883 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced-webhook-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:58:11.716262 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.716230 4883 generic.go:356] "Generic (PLEG): container finished" podID="3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced" containerID="4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01" exitCode=0 Oct 30 17:58:11.716698 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.716306 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689bcdb59-997v7" Oct 30 17:58:11.716698 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.716306 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689bcdb59-997v7" event={"ID":"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced","Type":"ContainerDied","Data":"4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01"} Oct 30 17:58:11.716698 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.716424 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689bcdb59-997v7" event={"ID":"3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced","Type":"ContainerDied","Data":"146ba3aeeefe36fa17eabc074a5d48b57fe2970a24a34d9458ea6167de5d3c4e"} Oct 30 17:58:11.716698 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.716452 4883 scope.go:122] "RemoveContainer" containerID="4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01" Oct 30 17:58:11.727297 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.727278 4883 scope.go:122] "RemoveContainer" containerID="4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01" Oct 30 17:58:11.727607 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:58:11.727588 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01\": container with ID starting with 4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01 not found: ID does not exist" containerID="4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01" Oct 30 17:58:11.727676 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.727620 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01"} err="failed to get container status \"4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01\": rpc error: code = NotFound desc = could not find container \"4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01\": container with ID starting with 4a2fe9cdfbd23b69b9bcb1ef5287bca886356b315bd7408fe5d3711958265a01 not found: ID does not exist" Oct 30 17:58:11.772620 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.772548 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8689bcdb59-997v7"] Oct 30 17:58:11.785290 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:11.785252 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8689bcdb59-997v7"] Oct 30 17:58:13.178022 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:13.177982 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced" path="/var/lib/kubelet/pods/3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced/volumes" Oct 30 17:58:14.733598 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:14.733560 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" event={"ID":"eee8dbb1-8a5b-4600-893d-78ecd9b403fe","Type":"ContainerStarted","Data":"a5da523085c33c8e416b9bd5832fd9d71fc4c879159c1ba8e24ab1d05ebe8bbd"} Oct 30 17:58:14.734098 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:14.733690 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:14.779133 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:14.779076 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" podStartSLOduration=2.470463982 podStartE2EDuration="13.779057681s" totalImagesPullingTime="11.308593699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:14.777334881 +0000 UTC m=+353.734338717" watchObservedRunningTime="2026-10-30 17:58:14.779057681 +0000 UTC m=+353.736061517" Oct 30 17:58:21.719075 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:21.719027 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-d66qd" Oct 30 17:58:22.038082 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.038002 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp"] Oct 30 17:58:22.038392 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.038380 4883 cpu_manager.go:414] "containerMap: removing container" podUID="3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced" containerName="manager" Oct 30 17:58:22.038433 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.038394 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced" containerName="manager" Oct 30 17:58:22.038472 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.038463 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8b52b1-5b81-4db9-93aa-0e1f32eb2ced" containerName="manager" Oct 30 17:58:22.042648 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.042633 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.046300 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.046270 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9389fe6d-d70c-45ec-8703-1dda36352377-metrics-certs\") pod \"jobset-controller-manager-68b94c7695-hqsdp\" (UID: \"9389fe6d-d70c-45ec-8703-1dda36352377\") " pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.046449 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.046428 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9389fe6d-d70c-45ec-8703-1dda36352377-cert\") pod \"jobset-controller-manager-68b94c7695-hqsdp\" (UID: \"9389fe6d-d70c-45ec-8703-1dda36352377\") " pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.046518 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.046475 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9knvt\" (UniqueName: \"kubernetes.io/projected/9389fe6d-d70c-45ec-8703-1dda36352377-kube-api-access-9knvt\") pod \"jobset-controller-manager-68b94c7695-hqsdp\" (UID: \"9389fe6d-d70c-45ec-8703-1dda36352377\") " pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.046573 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.046516 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9389fe6d-d70c-45ec-8703-1dda36352377-manager-config\") pod \"jobset-controller-manager-68b94c7695-hqsdp\" (UID: \"9389fe6d-d70c-45ec-8703-1dda36352377\") " pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.061432 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.061410 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.064780 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.064760 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.064952 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.064805 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 17:58:22.078345 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.078322 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-xxfsr\"" Oct 30 17:58:22.078469 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.078366 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 17:58:22.078469 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.078387 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 17:58:22.080974 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.080954 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp"] Oct 30 17:58:22.146988 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.146923 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9389fe6d-d70c-45ec-8703-1dda36352377-cert\") pod \"jobset-controller-manager-68b94c7695-hqsdp\" (UID: \"9389fe6d-d70c-45ec-8703-1dda36352377\") " pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.147162 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.146997 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9knvt\" (UniqueName: \"kubernetes.io/projected/9389fe6d-d70c-45ec-8703-1dda36352377-kube-api-access-9knvt\") pod \"jobset-controller-manager-68b94c7695-hqsdp\" (UID: \"9389fe6d-d70c-45ec-8703-1dda36352377\") " pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.147162 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.147022 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9389fe6d-d70c-45ec-8703-1dda36352377-manager-config\") pod \"jobset-controller-manager-68b94c7695-hqsdp\" (UID: \"9389fe6d-d70c-45ec-8703-1dda36352377\") " pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.147162 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.147047 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9389fe6d-d70c-45ec-8703-1dda36352377-metrics-certs\") pod \"jobset-controller-manager-68b94c7695-hqsdp\" (UID: \"9389fe6d-d70c-45ec-8703-1dda36352377\") " pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.147783 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.147756 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9389fe6d-d70c-45ec-8703-1dda36352377-manager-config\") pod \"jobset-controller-manager-68b94c7695-hqsdp\" (UID: \"9389fe6d-d70c-45ec-8703-1dda36352377\") " pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.149453 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.149430 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9389fe6d-d70c-45ec-8703-1dda36352377-cert\") pod \"jobset-controller-manager-68b94c7695-hqsdp\" (UID: \"9389fe6d-d70c-45ec-8703-1dda36352377\") " pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.149540 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.149455 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9389fe6d-d70c-45ec-8703-1dda36352377-metrics-certs\") pod \"jobset-controller-manager-68b94c7695-hqsdp\" (UID: \"9389fe6d-d70c-45ec-8703-1dda36352377\") " pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.172237 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.172211 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9knvt\" (UniqueName: \"kubernetes.io/projected/9389fe6d-d70c-45ec-8703-1dda36352377-kube-api-access-9knvt\") pod \"jobset-controller-manager-68b94c7695-hqsdp\" (UID: \"9389fe6d-d70c-45ec-8703-1dda36352377\") " pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.353926 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.353899 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:22.654520 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.654446 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp"] Oct 30 17:58:22.657648 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:58:22.657620 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9389fe6d_d70c_45ec_8703_1dda36352377.slice/crio-940ff5e8c111caf259de1b66eb3876ea2dc3d1a467cf1af27bba54c191d1f093 WatchSource:0}: Error finding container 940ff5e8c111caf259de1b66eb3876ea2dc3d1a467cf1af27bba54c191d1f093: Status 404 returned error can't find the container with id 940ff5e8c111caf259de1b66eb3876ea2dc3d1a467cf1af27bba54c191d1f093 Oct 30 17:58:22.769761 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:22.769725 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" event={"ID":"9389fe6d-d70c-45ec-8703-1dda36352377","Type":"ContainerStarted","Data":"940ff5e8c111caf259de1b66eb3876ea2dc3d1a467cf1af27bba54c191d1f093"} Oct 30 17:58:25.744368 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:25.744334 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xz9f" Oct 30 17:58:25.787432 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:25.787399 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" event={"ID":"9389fe6d-d70c-45ec-8703-1dda36352377","Type":"ContainerStarted","Data":"4d9be54b6bceca4ffccf08ffbcdf7d2595b09f20726832bcba2a5ac03f1294a1"} Oct 30 17:58:25.787606 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:25.787489 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:25.834673 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:25.834617 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" podStartSLOduration=2.250097857 podStartE2EDuration="4.834601907s" totalImagesPullingTime="2.58450405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:25.83339083 +0000 UTC m=+364.790394668" watchObservedRunningTime="2026-10-30 17:58:25.834601907 +0000 UTC m=+364.791605743" Oct 30 17:58:35.081084 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:58:35.081044 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:58:36.797174 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:36.797142 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-68b94c7695-hqsdp" Oct 30 17:58:40.710778 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:40.710746 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" Oct 30 17:58:41.797381 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:41.797350 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76c7f55f9d-d5v59"] Oct 30 17:58:41.930046 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:41.930013 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76c7f55f9d-d5v59"] Oct 30 17:58:41.930205 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:41.930134 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76c7f55f9d-d5v59" Oct 30 17:58:41.946447 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:41.946421 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 17:58:42.012761 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:42.012724 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mng8c\" (UniqueName: \"kubernetes.io/projected/34f343ed-77cc-4dad-b9ef-4200dd202187-kube-api-access-mng8c\") pod \"authorino-76c7f55f9d-d5v59\" (UID: \"34f343ed-77cc-4dad-b9ef-4200dd202187\") " pod="kuadrant-system/authorino-76c7f55f9d-d5v59" Oct 30 17:58:42.012761 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:42.012761 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/34f343ed-77cc-4dad-b9ef-4200dd202187-tls-cert\") pod \"authorino-76c7f55f9d-d5v59\" (UID: \"34f343ed-77cc-4dad-b9ef-4200dd202187\") " pod="kuadrant-system/authorino-76c7f55f9d-d5v59" Oct 30 17:58:42.113320 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:42.113283 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/34f343ed-77cc-4dad-b9ef-4200dd202187-tls-cert\") pod \"authorino-76c7f55f9d-d5v59\" (UID: \"34f343ed-77cc-4dad-b9ef-4200dd202187\") " pod="kuadrant-system/authorino-76c7f55f9d-d5v59" Oct 30 17:58:42.113481 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:42.113395 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mng8c\" (UniqueName: \"kubernetes.io/projected/34f343ed-77cc-4dad-b9ef-4200dd202187-kube-api-access-mng8c\") pod \"authorino-76c7f55f9d-d5v59\" (UID: \"34f343ed-77cc-4dad-b9ef-4200dd202187\") " pod="kuadrant-system/authorino-76c7f55f9d-d5v59" Oct 30 17:58:42.115707 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:42.115688 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/34f343ed-77cc-4dad-b9ef-4200dd202187-tls-cert\") pod \"authorino-76c7f55f9d-d5v59\" (UID: \"34f343ed-77cc-4dad-b9ef-4200dd202187\") " pod="kuadrant-system/authorino-76c7f55f9d-d5v59" Oct 30 17:58:42.137970 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:42.137944 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mng8c\" (UniqueName: \"kubernetes.io/projected/34f343ed-77cc-4dad-b9ef-4200dd202187-kube-api-access-mng8c\") pod \"authorino-76c7f55f9d-d5v59\" (UID: \"34f343ed-77cc-4dad-b9ef-4200dd202187\") " pod="kuadrant-system/authorino-76c7f55f9d-d5v59" Oct 30 17:58:42.257219 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:42.257182 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76c7f55f9d-d5v59" Oct 30 17:58:42.566650 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:42.566567 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76c7f55f9d-d5v59"] Oct 30 17:58:42.570089 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:58:42.570061 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34f343ed_77cc_4dad_b9ef_4200dd202187.slice/crio-02aaf52a6219b0d02812fc4b76df7583795431738d69ef677eca569b8a569fcb WatchSource:0}: Error finding container 02aaf52a6219b0d02812fc4b76df7583795431738d69ef677eca569b8a569fcb: Status 404 returned error can't find the container with id 02aaf52a6219b0d02812fc4b76df7583795431738d69ef677eca569b8a569fcb Oct 30 17:58:42.871227 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:42.871193 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76c7f55f9d-d5v59" event={"ID":"34f343ed-77cc-4dad-b9ef-4200dd202187","Type":"ContainerStarted","Data":"02aaf52a6219b0d02812fc4b76df7583795431738d69ef677eca569b8a569fcb"} Oct 30 17:58:43.213223 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:43.213195 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76c7f55f9d-d5v59"] Oct 30 17:58:43.880439 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:43.879177 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76c7f55f9d-d5v59" event={"ID":"34f343ed-77cc-4dad-b9ef-4200dd202187","Type":"ContainerStarted","Data":"2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd"} Oct 30 17:58:43.880439 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:43.879259 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-76c7f55f9d-d5v59" podUID="34f343ed-77cc-4dad-b9ef-4200dd202187" containerName="authorino" containerID="cri-o://2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd" gracePeriod=30 Oct 30 17:58:43.924387 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:43.924319 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-76c7f55f9d-d5v59" podStartSLOduration=2.30107724 podStartE2EDuration="2.924297107s" totalImagesPullingTime="623.219867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:43.921651342 +0000 UTC m=+382.878655179" watchObservedRunningTime="2026-10-30 17:58:43.924297107 +0000 UTC m=+382.881300944" Oct 30 17:58:44.165728 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.165693 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76c7f55f9d-d5v59" Oct 30 17:58:44.234712 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.234678 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f343ed-77cc-4dad-b9ef-4200dd202187-kube-api-access-mng8c\" (UniqueName: \"kubernetes.io/projected/34f343ed-77cc-4dad-b9ef-4200dd202187-kube-api-access-mng8c\") pod \"34f343ed-77cc-4dad-b9ef-4200dd202187\" (UID: \"34f343ed-77cc-4dad-b9ef-4200dd202187\") " Oct 30 17:58:44.234901 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.234730 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34f343ed-77cc-4dad-b9ef-4200dd202187-tls-cert\" (UniqueName: \"kubernetes.io/secret/34f343ed-77cc-4dad-b9ef-4200dd202187-tls-cert\") pod \"34f343ed-77cc-4dad-b9ef-4200dd202187\" (UID: \"34f343ed-77cc-4dad-b9ef-4200dd202187\") " Oct 30 17:58:44.236912 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.236873 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f343ed-77cc-4dad-b9ef-4200dd202187-kube-api-access-mng8c" pod "34f343ed-77cc-4dad-b9ef-4200dd202187" (UID: "34f343ed-77cc-4dad-b9ef-4200dd202187"). InnerVolumeSpecName "kube-api-access-mng8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:44.244918 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.244893 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f343ed-77cc-4dad-b9ef-4200dd202187-tls-cert" pod "34f343ed-77cc-4dad-b9ef-4200dd202187" (UID: "34f343ed-77cc-4dad-b9ef-4200dd202187"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:44.336257 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.336223 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mng8c\" (UniqueName: \"kubernetes.io/projected/34f343ed-77cc-4dad-b9ef-4200dd202187-kube-api-access-mng8c\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:58:44.336257 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.336250 4883 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/34f343ed-77cc-4dad-b9ef-4200dd202187-tls-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:58:44.855005 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.854922 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dc8775675-n5vjm"] Oct 30 17:58:44.855227 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.855197 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-dc8775675-n5vjm" podUID="d4b3ed18-17ff-44c8-b38d-c67730d0f6c6" containerName="authorino" containerID="cri-o://9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f" gracePeriod=30 Oct 30 17:58:44.885365 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.885329 4883 generic.go:356] "Generic (PLEG): container finished" podID="34f343ed-77cc-4dad-b9ef-4200dd202187" containerID="2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd" exitCode=0 Oct 30 17:58:44.885796 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.885394 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76c7f55f9d-d5v59" Oct 30 17:58:44.885796 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.885385 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76c7f55f9d-d5v59" event={"ID":"34f343ed-77cc-4dad-b9ef-4200dd202187","Type":"ContainerDied","Data":"2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd"} Oct 30 17:58:44.885796 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.885530 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76c7f55f9d-d5v59" event={"ID":"34f343ed-77cc-4dad-b9ef-4200dd202187","Type":"ContainerDied","Data":"02aaf52a6219b0d02812fc4b76df7583795431738d69ef677eca569b8a569fcb"} Oct 30 17:58:44.885796 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.885559 4883 scope.go:122] "RemoveContainer" containerID="2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd" Oct 30 17:58:44.897718 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.897659 4883 scope.go:122] "RemoveContainer" containerID="2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd" Oct 30 17:58:44.898226 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:58:44.898199 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd\": container with ID starting with 2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd not found: ID does not exist" containerID="2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd" Oct 30 17:58:44.898602 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.898238 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd"} err="failed to get container status \"2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd\": rpc error: code = NotFound desc = could not find container \"2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd\": container with ID starting with 2e2bd69de11b215b9818fa3c0dde8cfe2c358c775d90a767dc40b8e5a3fec0cd not found: ID does not exist" Oct 30 17:58:44.956562 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.956528 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76c7f55f9d-d5v59"] Oct 30 17:58:44.960886 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:44.960855 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76c7f55f9d-d5v59"] Oct 30 17:58:45.134994 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.134966 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc8775675-n5vjm" Oct 30 17:58:45.176137 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.176095 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f343ed-77cc-4dad-b9ef-4200dd202187" path="/var/lib/kubelet/pods/34f343ed-77cc-4dad-b9ef-4200dd202187/volumes" Oct 30 17:58:45.258349 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.258313 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b3ed18-17ff-44c8-b38d-c67730d0f6c6-kube-api-access-v56g5\" (UniqueName: \"kubernetes.io/projected/d4b3ed18-17ff-44c8-b38d-c67730d0f6c6-kube-api-access-v56g5\") pod \"d4b3ed18-17ff-44c8-b38d-c67730d0f6c6\" (UID: \"d4b3ed18-17ff-44c8-b38d-c67730d0f6c6\") " Oct 30 17:58:45.260431 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.260405 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b3ed18-17ff-44c8-b38d-c67730d0f6c6-kube-api-access-v56g5" pod "d4b3ed18-17ff-44c8-b38d-c67730d0f6c6" (UID: "d4b3ed18-17ff-44c8-b38d-c67730d0f6c6"). InnerVolumeSpecName "kube-api-access-v56g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:45.359291 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.359205 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v56g5\" (UniqueName: \"kubernetes.io/projected/d4b3ed18-17ff-44c8-b38d-c67730d0f6c6-kube-api-access-v56g5\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:58:45.892637 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.892599 4883 generic.go:356] "Generic (PLEG): container finished" podID="d4b3ed18-17ff-44c8-b38d-c67730d0f6c6" containerID="9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f" exitCode=0 Oct 30 17:58:45.893104 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.892640 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc8775675-n5vjm" Oct 30 17:58:45.893104 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.892681 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc8775675-n5vjm" event={"ID":"d4b3ed18-17ff-44c8-b38d-c67730d0f6c6","Type":"ContainerDied","Data":"9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f"} Oct 30 17:58:45.893104 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.892718 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc8775675-n5vjm" event={"ID":"d4b3ed18-17ff-44c8-b38d-c67730d0f6c6","Type":"ContainerDied","Data":"c653070b31b90009bdae64c9b279f651a55329b05eaeb9ab58738a292a6a01aa"} Oct 30 17:58:45.893104 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.892735 4883 scope.go:122] "RemoveContainer" containerID="9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f" Oct 30 17:58:45.901462 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.901447 4883 scope.go:122] "RemoveContainer" containerID="9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f" Oct 30 17:58:45.901724 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:58:45.901708 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f\": container with ID starting with 9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f not found: ID does not exist" containerID="9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f" Oct 30 17:58:45.901767 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.901736 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f"} err="failed to get container status \"9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f\": rpc error: code = NotFound desc = could not find container \"9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f\": container with ID starting with 9f6645943e4fd5be807bb42f96eb74618bbbee8679ee630841b30ebe098b133f not found: ID does not exist" Oct 30 17:58:45.947296 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.947252 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dc8775675-n5vjm"] Oct 30 17:58:45.951644 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:45.951623 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-dc8775675-n5vjm"] Oct 30 17:58:47.176762 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:58:47.176728 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b3ed18-17ff-44c8-b38d-c67730d0f6c6" path="/var/lib/kubelet/pods/d4b3ed18-17ff-44c8-b38d-c67730d0f6c6/volumes" Oct 30 17:59:05.077248 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:59:05.077170 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:59:05.980704 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:05.980664 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4"] Oct 30 17:59:05.981257 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:05.981236 4883 cpu_manager.go:414] "containerMap: removing container" podUID="34f343ed-77cc-4dad-b9ef-4200dd202187" containerName="authorino" Oct 30 17:59:05.981355 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:05.981258 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f343ed-77cc-4dad-b9ef-4200dd202187" containerName="authorino" Oct 30 17:59:05.981355 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:05.981278 4883 cpu_manager.go:414] "containerMap: removing container" podUID="d4b3ed18-17ff-44c8-b38d-c67730d0f6c6" containerName="authorino" Oct 30 17:59:05.981355 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:05.981286 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b3ed18-17ff-44c8-b38d-c67730d0f6c6" containerName="authorino" Oct 30 17:59:05.981501 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:05.981413 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f343ed-77cc-4dad-b9ef-4200dd202187" containerName="authorino" Oct 30 17:59:05.981501 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:05.981424 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b3ed18-17ff-44c8-b38d-c67730d0f6c6" containerName="authorino" Oct 30 17:59:05.984760 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:05.984736 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" Oct 30 17:59:06.003638 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:06.003613 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4"] Oct 30 17:59:06.164639 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:06.164603 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr9qr\" (UniqueName: \"kubernetes.io/projected/016abd83-e950-417f-8778-657aa6fe09a6-kube-api-access-kr9qr\") pod \"llmisvc-controller-manager-6466c554d8-lw4w4\" (UID: \"016abd83-e950-417f-8778-657aa6fe09a6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" Oct 30 17:59:06.165070 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:06.164663 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/016abd83-e950-417f-8778-657aa6fe09a6-cert\") pod \"llmisvc-controller-manager-6466c554d8-lw4w4\" (UID: \"016abd83-e950-417f-8778-657aa6fe09a6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" Oct 30 17:59:06.266167 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:06.266086 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9qr\" (UniqueName: \"kubernetes.io/projected/016abd83-e950-417f-8778-657aa6fe09a6-kube-api-access-kr9qr\") pod \"llmisvc-controller-manager-6466c554d8-lw4w4\" (UID: \"016abd83-e950-417f-8778-657aa6fe09a6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" Oct 30 17:59:06.266167 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:06.266138 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/016abd83-e950-417f-8778-657aa6fe09a6-cert\") pod \"llmisvc-controller-manager-6466c554d8-lw4w4\" (UID: \"016abd83-e950-417f-8778-657aa6fe09a6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" Oct 30 17:59:06.268551 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:06.268528 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/016abd83-e950-417f-8778-657aa6fe09a6-cert\") pod \"llmisvc-controller-manager-6466c554d8-lw4w4\" (UID: \"016abd83-e950-417f-8778-657aa6fe09a6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" Oct 30 17:59:06.291170 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:06.291141 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9qr\" (UniqueName: \"kubernetes.io/projected/016abd83-e950-417f-8778-657aa6fe09a6-kube-api-access-kr9qr\") pod \"llmisvc-controller-manager-6466c554d8-lw4w4\" (UID: \"016abd83-e950-417f-8778-657aa6fe09a6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" Oct 30 17:59:06.301018 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:06.300995 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" Oct 30 17:59:06.600956 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:06.600915 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4"] Oct 30 17:59:06.604743 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:59:06.604714 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016abd83_e950_417f_8778_657aa6fe09a6.slice/crio-bc304c884d48f69011cdb1776d0c3ea22db195ceb64e99a47589e5ea0adce0ad WatchSource:0}: Error finding container bc304c884d48f69011cdb1776d0c3ea22db195ceb64e99a47589e5ea0adce0ad: Status 404 returned error can't find the container with id bc304c884d48f69011cdb1776d0c3ea22db195ceb64e99a47589e5ea0adce0ad Oct 30 17:59:06.988725 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:06.988699 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" event={"ID":"016abd83-e950-417f-8778-657aa6fe09a6","Type":"ContainerStarted","Data":"bc304c884d48f69011cdb1776d0c3ea22db195ceb64e99a47589e5ea0adce0ad"} Oct 30 17:59:07.994298 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:07.994261 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" event={"ID":"016abd83-e950-417f-8778-657aa6fe09a6","Type":"ContainerStarted","Data":"5bbc5e356b05ef081c3b6dbb88973bfc6d18ad6a24c266308e9c69c1dc1df316"} Oct 30 17:59:07.994757 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:07.994433 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" Oct 30 17:59:08.038325 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:08.037881 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" podStartSLOduration=2.696388512 podStartE2EDuration="3.037864063s" totalImagesPullingTime="341.475551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:08.037759988 +0000 UTC m=+406.994763825" watchObservedRunningTime="2026-10-30 17:59:08.037864063 +0000 UTC m=+406.994867903" Oct 30 17:59:35.077082 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:59:35.077042 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 17:59:38.105235 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.105194 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x"] Oct 30 17:59:38.109963 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.109573 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.129978 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.129924 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 17:59:38.144349 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.144318 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x"] Oct 30 17:59:38.212184 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.212158 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kj7cx\"" Oct 30 17:59:38.219515 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.219487 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18290ea1-6092-4b79-b48e-40121387ad50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.219582 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.219524 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6z5\" (UniqueName: \"kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-kube-api-access-5l6z5\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.219649 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.219637 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.219683 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.219658 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.219749 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.219737 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.219782 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.219759 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.219817 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.219783 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.219880 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.219867 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.219911 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.219890 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18290ea1-6092-4b79-b48e-40121387ad50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321069 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321034 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18290ea1-6092-4b79-b48e-40121387ad50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321237 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321086 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18290ea1-6092-4b79-b48e-40121387ad50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321237 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321203 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6z5\" (UniqueName: \"kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-kube-api-access-5l6z5\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321323 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321307 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321360 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321337 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321416 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321372 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321416 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321403 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321518 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321436 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321518 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321485 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321784 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321729 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321925 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321841 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321925 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321847 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18290ea1-6092-4b79-b48e-40121387ad50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321925 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321891 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.321925 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.321928 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.323610 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.323583 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.323728 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.323713 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18290ea1-6092-4b79-b48e-40121387ad50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.347010 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.346904 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.347010 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.347000 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6z5\" (UniqueName: \"kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-kube-api-access-5l6z5\") pod \"maas-default-gateway-openshift-default-67bf986f99-p8z8x\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.425808 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.425727 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:38.745504 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.745455 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x"] Oct 30 17:59:38.747946 ip-10-0-130-250 kubenswrapper[4883]: W1030 17:59:38.747901 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18290ea1_6092_4b79_b48e_40121387ad50.slice/crio-bf76e315f0150e7cd015d947f6c7874d155aa4c2d55be6cf20ef2c2854fa066f WatchSource:0}: Error finding container bf76e315f0150e7cd015d947f6c7874d155aa4c2d55be6cf20ef2c2854fa066f: Status 404 returned error can't find the container with id bf76e315f0150e7cd015d947f6c7874d155aa4c2d55be6cf20ef2c2854fa066f Oct 30 17:59:38.999362 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:38.999283 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lw4w4" Oct 30 17:59:39.074216 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:39.074178 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp"] Oct 30 17:59:39.074433 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:39.074407 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" podUID="764b4161-26c5-4cba-a529-619ccef2bdab" containerName="manager" containerID="cri-o://9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2" gracePeriod=30 Oct 30 17:59:39.134013 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:39.133972 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" event={"ID":"18290ea1-6092-4b79-b48e-40121387ad50","Type":"ContainerStarted","Data":"bf76e315f0150e7cd015d947f6c7874d155aa4c2d55be6cf20ef2c2854fa066f"} Oct 30 17:59:39.316713 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:39.316686 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" Oct 30 17:59:39.430865 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:39.430831 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764b4161-26c5-4cba-a529-619ccef2bdab-cert\" (UniqueName: \"kubernetes.io/secret/764b4161-26c5-4cba-a529-619ccef2bdab-cert\") pod \"764b4161-26c5-4cba-a529-619ccef2bdab\" (UID: \"764b4161-26c5-4cba-a529-619ccef2bdab\") " Oct 30 17:59:39.431059 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:39.430973 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764b4161-26c5-4cba-a529-619ccef2bdab-kube-api-access-ghr88\" (UniqueName: \"kubernetes.io/projected/764b4161-26c5-4cba-a529-619ccef2bdab-kube-api-access-ghr88\") pod \"764b4161-26c5-4cba-a529-619ccef2bdab\" (UID: \"764b4161-26c5-4cba-a529-619ccef2bdab\") " Oct 30 17:59:39.433438 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:39.433395 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764b4161-26c5-4cba-a529-619ccef2bdab-cert" pod "764b4161-26c5-4cba-a529-619ccef2bdab" (UID: "764b4161-26c5-4cba-a529-619ccef2bdab"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:39.433549 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:39.433507 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764b4161-26c5-4cba-a529-619ccef2bdab-kube-api-access-ghr88" pod "764b4161-26c5-4cba-a529-619ccef2bdab" (UID: "764b4161-26c5-4cba-a529-619ccef2bdab"). InnerVolumeSpecName "kube-api-access-ghr88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:39.532836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:39.532793 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghr88\" (UniqueName: \"kubernetes.io/projected/764b4161-26c5-4cba-a529-619ccef2bdab-kube-api-access-ghr88\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:59:39.532836 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:39.532836 4883 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/764b4161-26c5-4cba-a529-619ccef2bdab-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 17:59:40.142361 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:40.142322 4883 generic.go:356] "Generic (PLEG): container finished" podID="764b4161-26c5-4cba-a529-619ccef2bdab" containerID="9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2" exitCode=0 Oct 30 17:59:40.142826 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:40.142501 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" event={"ID":"764b4161-26c5-4cba-a529-619ccef2bdab","Type":"ContainerDied","Data":"9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2"} Oct 30 17:59:40.142826 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:40.142543 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" event={"ID":"764b4161-26c5-4cba-a529-619ccef2bdab","Type":"ContainerDied","Data":"bafdab0b10a848348121e26170aea83ac4a426f570203acc7a10d1060433df0b"} Oct 30 17:59:40.142826 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:40.142568 4883 scope.go:122] "RemoveContainer" containerID="9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2" Oct 30 17:59:40.142826 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:40.142691 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp" Oct 30 17:59:40.156646 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:40.156180 4883 scope.go:122] "RemoveContainer" containerID="9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2" Oct 30 17:59:40.156646 ip-10-0-130-250 kubenswrapper[4883]: E1030 17:59:40.156484 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2\": container with ID starting with 9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2 not found: ID does not exist" containerID="9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2" Oct 30 17:59:40.156646 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:40.156521 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2"} err="failed to get container status \"9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2\": rpc error: code = NotFound desc = could not find container \"9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2\": container with ID starting with 9d2fbd44dbbf20adb0be5d8a83e8c4ee127b3d6a9491b890e2879929746a16f2 not found: ID does not exist" Oct 30 17:59:40.202966 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:40.202917 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp"] Oct 30 17:59:40.212602 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:40.212563 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gtngp"] Oct 30 17:59:41.177342 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:41.177309 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764b4161-26c5-4cba-a529-619ccef2bdab" path="/var/lib/kubelet/pods/764b4161-26c5-4cba-a529-619ccef2bdab/volumes" Oct 30 17:59:41.308595 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:41.308562 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 17:59:41.308679 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:41.308649 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 17:59:41.308726 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:41.308683 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 17:59:42.154495 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:42.154461 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" event={"ID":"18290ea1-6092-4b79-b48e-40121387ad50","Type":"ContainerStarted","Data":"4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48"} Oct 30 17:59:42.235468 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:42.235421 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" podStartSLOduration=1.677591434 podStartE2EDuration="4.235404914s" totalImagesPullingTime="2.55781348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:42.221192922 +0000 UTC m=+441.178196781" watchObservedRunningTime="2026-10-30 17:59:42.235404914 +0000 UTC m=+441.192408750" Oct 30 17:59:42.426256 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:42.426169 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:42.427837 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:42.427809 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-p8z8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 17:59:42.427973 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:42.427858 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" podUID="18290ea1-6092-4b79-b48e-40121387ad50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 17:59:43.426951 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:43.426909 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-p8z8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 17:59:43.427361 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:43.426983 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" podUID="18290ea1-6092-4b79-b48e-40121387ad50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 17:59:44.426736 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:44.426696 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-p8z8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 17:59:44.426899 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:44.426765 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" podUID="18290ea1-6092-4b79-b48e-40121387ad50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 17:59:45.426333 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:45.426296 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-p8z8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 17:59:45.426723 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:45.426362 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" podUID="18290ea1-6092-4b79-b48e-40121387ad50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 17:59:46.426617 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:46.426581 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-p8z8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 17:59:46.427037 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:46.426661 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" podUID="18290ea1-6092-4b79-b48e-40121387ad50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 17:59:47.623629 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:47.623576 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:47.624285 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:47.624262 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 17:59:47.625487 ip-10-0-130-250 kubenswrapper[4883]: I1030 17:59:47.625461 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 18:00:05.077471 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:00:05.077429 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:00:35.077046 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:00:35.076964 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:01:05.077708 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:01:05.077666 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:01:08.362811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.362771 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh"] Oct 30 18:01:08.363373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.363228 4883 cpu_manager.go:414] "containerMap: removing container" podUID="764b4161-26c5-4cba-a529-619ccef2bdab" containerName="manager" Oct 30 18:01:08.363373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.363242 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764b4161-26c5-4cba-a529-619ccef2bdab" containerName="manager" Oct 30 18:01:08.363373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.363360 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="764b4161-26c5-4cba-a529-619ccef2bdab" containerName="manager" Oct 30 18:01:08.366286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.366266 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.400066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.400036 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh"] Oct 30 18:01:08.404129 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.404105 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh"] Oct 30 18:01:08.404269 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.404203 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.423759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.423724 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh"] Oct 30 18:01:08.456562 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.456527 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.456735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.456576 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.456735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.456595 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhrfz\" (UniqueName: \"kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-kube-api-access-bhrfz\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.456735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.456719 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.456881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.456754 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.456881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.456780 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.456881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.456865 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.457115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.456881 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5beef11-370d-4e74-8d12-e0541cb2a653-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.457115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.456913 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.557778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.557735 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.558002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.557820 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.558002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.557859 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.558002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.557923 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.558002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.557972 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.558002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.557995 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrfz\" (UniqueName: \"kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-kube-api-access-bhrfz\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.558251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558025 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.558251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558065 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.558251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558157 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.558394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558252 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.558394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558277 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.558394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558309 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.558394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558350 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.558576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558391 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbfqj\" (UniqueName: \"kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-kube-api-access-tbfqj\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.558576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558492 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.558576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558518 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5beef11-370d-4e74-8d12-e0541cb2a653-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.558576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558545 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.558783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558572 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.558783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558605 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.558783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558744 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.558889 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558800 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.558999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.558980 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.559134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.559116 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5beef11-370d-4e74-8d12-e0541cb2a653-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.560650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.560628 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.560736 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.560697 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.588036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.588010 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.588295 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.588273 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrfz\" (UniqueName: \"kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-kube-api-access-bhrfz\") pod \"maas-default-gateway-openshift-default-67bf986f99-cntbh\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.660030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.659920 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.660030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.659997 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.660266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.660053 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.660266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.660097 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.660266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.660129 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.660266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.660182 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.660266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.660207 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.660266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.660236 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.660266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.660265 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbfqj\" (UniqueName: \"kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-kube-api-access-tbfqj\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.660786 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.660762 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.660887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.660827 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.660994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.660973 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.661062 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.660995 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.661223 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.661201 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.662447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.662426 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.662835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.662818 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.679613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.679585 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:08.685493 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.685471 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbfqj\" (UniqueName: \"kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-kube-api-access-tbfqj\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.685784 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.685766 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cxsgh\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:08.718562 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:08.718518 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:09.000786 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.000708 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh"] Oct 30 18:01:09.003825 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:01:09.003782 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5beef11_370d_4e74_8d12_e0541cb2a653.slice/crio-4cb4362ed910171eda91b09cd61dcdc5fa6b141fa492a2771ae10aaede8327cf WatchSource:0}: Error finding container 4cb4362ed910171eda91b09cd61dcdc5fa6b141fa492a2771ae10aaede8327cf: Status 404 returned error can't find the container with id 4cb4362ed910171eda91b09cd61dcdc5fa6b141fa492a2771ae10aaede8327cf Oct 30 18:01:09.006125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.006079 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:01:09.006209 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.006162 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:01:09.006209 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.006203 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:01:09.060616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.060591 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh"] Oct 30 18:01:09.063293 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:01:09.063265 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1c611e_b4ee_499c_8c86_1409d8461ac7.slice/crio-c7c9f9ecc8f296c89b6e9b362d0841932228ed1cbae20a90aebcf0f8cf1d178c WatchSource:0}: Error finding container c7c9f9ecc8f296c89b6e9b362d0841932228ed1cbae20a90aebcf0f8cf1d178c: Status 404 returned error can't find the container with id c7c9f9ecc8f296c89b6e9b362d0841932228ed1cbae20a90aebcf0f8cf1d178c Oct 30 18:01:09.065539 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.065506 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:01:09.065607 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.065585 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:01:09.065643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.065628 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:01:09.585281 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.585236 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" event={"ID":"9c1c611e-b4ee-499c-8c86-1409d8461ac7","Type":"ContainerStarted","Data":"793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6"} Oct 30 18:01:09.585709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.585304 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" event={"ID":"9c1c611e-b4ee-499c-8c86-1409d8461ac7","Type":"ContainerStarted","Data":"c7c9f9ecc8f296c89b6e9b362d0841932228ed1cbae20a90aebcf0f8cf1d178c"} Oct 30 18:01:09.587325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.587295 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" event={"ID":"a5beef11-370d-4e74-8d12-e0541cb2a653","Type":"ContainerStarted","Data":"3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a"} Oct 30 18:01:09.587467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.587344 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" event={"ID":"a5beef11-370d-4e74-8d12-e0541cb2a653","Type":"ContainerStarted","Data":"4cb4362ed910171eda91b09cd61dcdc5fa6b141fa492a2771ae10aaede8327cf"} Oct 30 18:01:09.638457 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.638392 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" podStartSLOduration=1.63837363 podStartE2EDuration="1.63837363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:09.63781014 +0000 UTC m=+528.594813975" watchObservedRunningTime="2026-10-30 18:01:09.63837363 +0000 UTC m=+528.595377468" Oct 30 18:01:09.681064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.681029 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:09.726183 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:09.726151 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:10.684674 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:10.684636 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cntbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:01:10.685299 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:10.685268 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:01:10.721961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:10.721435 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cxsgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:01:10.721961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:10.721501 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:01:11.691958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:11.690585 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cntbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:01:11.691958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:11.690649 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:01:11.719454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:11.719417 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cxsgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:01:11.719732 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:11.719703 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:01:12.680817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:12.680358 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cntbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:01:12.680817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:12.680424 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:01:12.719062 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:12.719023 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cxsgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:01:12.719528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:12.719500 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:01:13.698016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:13.690367 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cntbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:01:13.698016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:13.690430 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:01:13.723477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:13.723354 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cxsgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:01:13.723477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:13.723414 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:01:14.686958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:14.686040 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cntbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:01:14.686958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:14.686105 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:01:14.720531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:14.720403 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cxsgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:01:14.720531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:14.720468 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:01:15.680722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:15.680674 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cntbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:01:15.681223 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:15.680767 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:01:15.719764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:15.719718 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cxsgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:01:15.719951 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:15.719804 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:01:16.680050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:16.680009 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cntbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:01:16.680227 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:16.680097 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:01:16.719902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:16.719858 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cxsgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:01:16.720273 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:16.719922 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 18:01:17.931159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:17.931110 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:17.931669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:17.931546 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:17.932747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:17.932725 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:01:17.946052 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:17.946012 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:17.946542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:17.946502 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:17.947203 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:17.947183 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:01:17.995164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:01:17.994410 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" podStartSLOduration=9.994393396 podStartE2EDuration="9.994393396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:09.693337148 +0000 UTC m=+528.650340984" watchObservedRunningTime="2026-10-30 18:01:17.994393396 +0000 UTC m=+536.951397233" Oct 30 18:01:35.077767 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:01:35.077726 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:02:05.078363 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:02:05.078281 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:02:21.241103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:02:21.241065 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:02:21.241854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:02:21.241832 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:02:21.247172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:02:21.247146 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:02:21.247904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:02:21.247886 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:02:35.077980 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:02:35.077907 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:03:05.079340 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:03:05.079303 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:03:24.777841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:24.777801 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl"] Oct 30 18:03:24.781728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:24.781705 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:24.800771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:24.800675 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:03:24.800771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:24.800736 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:03:24.801062 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:24.800853 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:03:24.803249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:24.803221 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl"] Oct 30 18:03:24.814050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:24.813995 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:03:24.909747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:24.909704 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4fp\" (UniqueName: \"kubernetes.io/projected/8856cb09-5a3c-4d33-80da-21dda441b2a1-kube-api-access-mp4fp\") pod \"maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl\" (UID: \"8856cb09-5a3c-4d33-80da-21dda441b2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:24.909973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:24.909835 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl\" (UID: \"8856cb09-5a3c-4d33-80da-21dda441b2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:24.909973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:24.909914 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl\" (UID: \"8856cb09-5a3c-4d33-80da-21dda441b2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:25.011085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:25.011038 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl\" (UID: \"8856cb09-5a3c-4d33-80da-21dda441b2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:25.011296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:25.011137 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl\" (UID: \"8856cb09-5a3c-4d33-80da-21dda441b2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:25.011296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:25.011176 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4fp\" (UniqueName: \"kubernetes.io/projected/8856cb09-5a3c-4d33-80da-21dda441b2a1-kube-api-access-mp4fp\") pod \"maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl\" (UID: \"8856cb09-5a3c-4d33-80da-21dda441b2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:25.011952 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:25.011917 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl\" (UID: \"8856cb09-5a3c-4d33-80da-21dda441b2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:25.013602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:25.013575 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl\" (UID: \"8856cb09-5a3c-4d33-80da-21dda441b2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:25.036490 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:25.036417 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4fp\" (UniqueName: \"kubernetes.io/projected/8856cb09-5a3c-4d33-80da-21dda441b2a1-kube-api-access-mp4fp\") pod \"maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl\" (UID: \"8856cb09-5a3c-4d33-80da-21dda441b2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:25.095668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:25.095629 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:25.402985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:25.402919 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl"] Oct 30 18:03:25.406337 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:03:25.406306 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8856cb09_5a3c_4d33_80da_21dda441b2a1.slice/crio-b31074b7c2aa3dfdb652493e3a9faa982ea3a80546cb29d3d96a8f021b9f00d5 WatchSource:0}: Error finding container b31074b7c2aa3dfdb652493e3a9faa982ea3a80546cb29d3d96a8f021b9f00d5: Status 404 returned error can't find the container with id b31074b7c2aa3dfdb652493e3a9faa982ea3a80546cb29d3d96a8f021b9f00d5 Oct 30 18:03:25.408604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:25.408585 4883 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:03:26.246245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:26.246197 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" event={"ID":"8856cb09-5a3c-4d33-80da-21dda441b2a1","Type":"ContainerStarted","Data":"b31074b7c2aa3dfdb652493e3a9faa982ea3a80546cb29d3d96a8f021b9f00d5"} Oct 30 18:03:28.258236 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:28.258133 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" event={"ID":"8856cb09-5a3c-4d33-80da-21dda441b2a1","Type":"ContainerStarted","Data":"c6f9bb5cc6809d726869a310b1c2192a79d5a8daaa2eb1e78804c4d0320085a9"} Oct 30 18:03:28.302053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:28.301998 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" podStartSLOduration=2.139391366 podStartE2EDuration="4.301981213s" totalImagesPullingTime="2.162589847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:28.300416177 +0000 UTC m=+667.257420012" watchObservedRunningTime="2026-10-30 18:03:28.301981213 +0000 UTC m=+667.258985049" Oct 30 18:03:29.264048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:29.264015 4883 generic.go:356] "Generic (PLEG): container finished" podID="8856cb09-5a3c-4d33-80da-21dda441b2a1" containerID="c6f9bb5cc6809d726869a310b1c2192a79d5a8daaa2eb1e78804c4d0320085a9" exitCode=0 Oct 30 18:03:29.264497 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:29.264112 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" event={"ID":"8856cb09-5a3c-4d33-80da-21dda441b2a1","Type":"ContainerDied","Data":"c6f9bb5cc6809d726869a310b1c2192a79d5a8daaa2eb1e78804c4d0320085a9"} Oct 30 18:03:29.965376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:29.965336 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k"] Oct 30 18:03:29.968853 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:29.968831 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:29.991622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:29.991566 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k"] Oct 30 18:03:30.064540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.064503 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k\" (UID: \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:30.064717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.064548 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k\" (UID: \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:30.064794 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.064765 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dqpc\" (UniqueName: \"kubernetes.io/projected/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-kube-api-access-6dqpc\") pod \"maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k\" (UID: \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:30.166032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.165990 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dqpc\" (UniqueName: \"kubernetes.io/projected/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-kube-api-access-6dqpc\") pod \"maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k\" (UID: \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:30.166220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.166035 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k\" (UID: \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:30.166220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.166160 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k\" (UID: \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:30.166785 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.166749 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k\" (UID: \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:30.168441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.168419 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k\" (UID: \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:30.190944 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.190902 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dqpc\" (UniqueName: \"kubernetes.io/projected/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-kube-api-access-6dqpc\") pod \"maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k\" (UID: \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:30.282103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.282070 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:30.413832 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.413802 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:30.569019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.568893 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-xks-ca\") pod \"8856cb09-5a3c-4d33-80da-21dda441b2a1\" (UID: \"8856cb09-5a3c-4d33-80da-21dda441b2a1\") " Oct 30 18:03:30.569019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.568965 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-openshift-ca\") pod \"8856cb09-5a3c-4d33-80da-21dda441b2a1\" (UID: \"8856cb09-5a3c-4d33-80da-21dda441b2a1\") " Oct 30 18:03:30.569252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.569115 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8856cb09-5a3c-4d33-80da-21dda441b2a1-kube-api-access-mp4fp\" (UniqueName: \"kubernetes.io/projected/8856cb09-5a3c-4d33-80da-21dda441b2a1-kube-api-access-mp4fp\") pod \"8856cb09-5a3c-4d33-80da-21dda441b2a1\" (UID: \"8856cb09-5a3c-4d33-80da-21dda441b2a1\") " Oct 30 18:03:30.569362 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.569329 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-openshift-ca" pod "8856cb09-5a3c-4d33-80da-21dda441b2a1" (UID: "8856cb09-5a3c-4d33-80da-21dda441b2a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:30.571281 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.571248 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-xks-ca" pod "8856cb09-5a3c-4d33-80da-21dda441b2a1" (UID: "8856cb09-5a3c-4d33-80da-21dda441b2a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:30.571398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.571372 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8856cb09-5a3c-4d33-80da-21dda441b2a1-kube-api-access-mp4fp" pod "8856cb09-5a3c-4d33-80da-21dda441b2a1" (UID: "8856cb09-5a3c-4d33-80da-21dda441b2a1"). InnerVolumeSpecName "kube-api-access-mp4fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:30.599703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.599665 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k"] Oct 30 18:03:30.602782 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:03:30.602750 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca0b20a_3101_4db4_9f65_87d88a8a89c2.slice/crio-fbf35593385c72d4d2f63e040b7fe5305584f136a46e974ebf5ffb399a9e3e85 WatchSource:0}: Error finding container fbf35593385c72d4d2f63e040b7fe5305584f136a46e974ebf5ffb399a9e3e85: Status 404 returned error can't find the container with id fbf35593385c72d4d2f63e040b7fe5305584f136a46e974ebf5ffb399a9e3e85 Oct 30 18:03:30.670476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.670441 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4fp\" (UniqueName: \"kubernetes.io/projected/8856cb09-5a3c-4d33-80da-21dda441b2a1-kube-api-access-mp4fp\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:03:30.670476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.670473 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:03:30.670476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:30.670482 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8856cb09-5a3c-4d33-80da-21dda441b2a1-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:03:31.275463 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.275421 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" event={"ID":"0ca0b20a-3101-4db4-9f65-87d88a8a89c2","Type":"ContainerStarted","Data":"e956e8e3737776215a6d33ac9029b6cc25b91f968275769bb6ee184dcaa0b0ef"} Oct 30 18:03:31.275687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.275482 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" event={"ID":"0ca0b20a-3101-4db4-9f65-87d88a8a89c2","Type":"ContainerStarted","Data":"fbf35593385c72d4d2f63e040b7fe5305584f136a46e974ebf5ffb399a9e3e85"} Oct 30 18:03:31.277043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.277021 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" Oct 30 18:03:31.277144 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.277052 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl" event={"ID":"8856cb09-5a3c-4d33-80da-21dda441b2a1","Type":"ContainerDied","Data":"b31074b7c2aa3dfdb652493e3a9faa982ea3a80546cb29d3d96a8f021b9f00d5"} Oct 30 18:03:31.277144 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.277095 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b31074b7c2aa3dfdb652493e3a9faa982ea3a80546cb29d3d96a8f021b9f00d5" Oct 30 18:03:31.319560 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.319506 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" podStartSLOduration=2.3194896529999998 podStartE2EDuration="2.319489653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:31.317573468 +0000 UTC m=+670.274577304" watchObservedRunningTime="2026-10-30 18:03:31.319489653 +0000 UTC m=+670.276493522" Oct 30 18:03:31.700050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.700011 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm"] Oct 30 18:03:31.700447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.700432 4883 cpu_manager.go:414] "containerMap: removing container" podUID="8856cb09-5a3c-4d33-80da-21dda441b2a1" containerName="revoke-keys" Oct 30 18:03:31.700494 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.700449 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8856cb09-5a3c-4d33-80da-21dda441b2a1" containerName="revoke-keys" Oct 30 18:03:31.700565 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.700543 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="8856cb09-5a3c-4d33-80da-21dda441b2a1" containerName="revoke-keys" Oct 30 18:03:31.703986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.703962 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:31.725232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.725013 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm"] Oct 30 18:03:31.781218 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.781174 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm\" (UID: \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:31.781374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.781241 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trjnl\" (UniqueName: \"kubernetes.io/projected/e5e0d420-e5f8-4eb6-8764-ddd26b284227-kube-api-access-trjnl\") pod \"maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm\" (UID: \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:31.781374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.781352 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm\" (UID: \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:31.881896 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.881856 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm\" (UID: \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:31.882088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.881999 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm\" (UID: \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:31.882088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.882069 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trjnl\" (UniqueName: \"kubernetes.io/projected/e5e0d420-e5f8-4eb6-8764-ddd26b284227-kube-api-access-trjnl\") pod \"maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm\" (UID: \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:31.882592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.882564 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm\" (UID: \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:31.884401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.884381 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm\" (UID: \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:31.906901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:31.906860 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trjnl\" (UniqueName: \"kubernetes.io/projected/e5e0d420-e5f8-4eb6-8764-ddd26b284227-kube-api-access-trjnl\") pod \"maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm\" (UID: \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:32.017099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:32.017009 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:32.282947 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:32.282843 4883 generic.go:356] "Generic (PLEG): container finished" podID="0ca0b20a-3101-4db4-9f65-87d88a8a89c2" containerID="e956e8e3737776215a6d33ac9029b6cc25b91f968275769bb6ee184dcaa0b0ef" exitCode=0 Oct 30 18:03:32.283097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:32.282947 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" event={"ID":"0ca0b20a-3101-4db4-9f65-87d88a8a89c2","Type":"ContainerDied","Data":"e956e8e3737776215a6d33ac9029b6cc25b91f968275769bb6ee184dcaa0b0ef"} Oct 30 18:03:32.318350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:32.318199 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm"] Oct 30 18:03:32.321143 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:03:32.321114 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e0d420_e5f8_4eb6_8764_ddd26b284227.slice/crio-77a370f904e0bcf7e7670bb85a0554eb582054c7bfad0ed7b2e5e7b06e256c8c WatchSource:0}: Error finding container 77a370f904e0bcf7e7670bb85a0554eb582054c7bfad0ed7b2e5e7b06e256c8c: Status 404 returned error can't find the container with id 77a370f904e0bcf7e7670bb85a0554eb582054c7bfad0ed7b2e5e7b06e256c8c Oct 30 18:03:33.290467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.290429 4883 generic.go:356] "Generic (PLEG): container finished" podID="e5e0d420-e5f8-4eb6-8764-ddd26b284227" containerID="658a997edab539c7ea053bdc4fd32a41be4f7c386f9ecabee884b0d47cd0613f" exitCode=0 Oct 30 18:03:33.290686 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.290516 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" event={"ID":"e5e0d420-e5f8-4eb6-8764-ddd26b284227","Type":"ContainerDied","Data":"658a997edab539c7ea053bdc4fd32a41be4f7c386f9ecabee884b0d47cd0613f"} Oct 30 18:03:33.290686 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.290577 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" event={"ID":"e5e0d420-e5f8-4eb6-8764-ddd26b284227","Type":"ContainerStarted","Data":"77a370f904e0bcf7e7670bb85a0554eb582054c7bfad0ed7b2e5e7b06e256c8c"} Oct 30 18:03:33.441482 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.441450 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:33.599582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.599493 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-openshift-ca\") pod \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\" (UID: \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\") " Oct 30 18:03:33.599582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.599565 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-kube-api-access-6dqpc\" (UniqueName: \"kubernetes.io/projected/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-kube-api-access-6dqpc\") pod \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\" (UID: \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\") " Oct 30 18:03:33.599786 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.599676 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-xks-ca\") pod \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\" (UID: \"0ca0b20a-3101-4db4-9f65-87d88a8a89c2\") " Oct 30 18:03:33.599902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.599876 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-openshift-ca" pod "0ca0b20a-3101-4db4-9f65-87d88a8a89c2" (UID: "0ca0b20a-3101-4db4-9f65-87d88a8a89c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:33.600022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.599993 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:03:33.601752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.601724 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-xks-ca" pod "0ca0b20a-3101-4db4-9f65-87d88a8a89c2" (UID: "0ca0b20a-3101-4db4-9f65-87d88a8a89c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:33.601846 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.601800 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-kube-api-access-6dqpc" pod "0ca0b20a-3101-4db4-9f65-87d88a8a89c2" (UID: "0ca0b20a-3101-4db4-9f65-87d88a8a89c2"). InnerVolumeSpecName "kube-api-access-6dqpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:33.700919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.700863 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dqpc\" (UniqueName: \"kubernetes.io/projected/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-kube-api-access-6dqpc\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:03:33.700919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:33.700906 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca0b20a-3101-4db4-9f65-87d88a8a89c2-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:03:34.296527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.296478 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" event={"ID":"0ca0b20a-3101-4db4-9f65-87d88a8a89c2","Type":"ContainerDied","Data":"fbf35593385c72d4d2f63e040b7fe5305584f136a46e974ebf5ffb399a9e3e85"} Oct 30 18:03:34.296741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.296551 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbf35593385c72d4d2f63e040b7fe5305584f136a46e974ebf5ffb399a9e3e85" Oct 30 18:03:34.296741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.296503 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k" Oct 30 18:03:34.442327 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.442294 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:34.610825 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.610782 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-openshift-ca\") pod \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\" (UID: \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\") " Oct 30 18:03:34.611056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.610877 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-xks-ca\") pod \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\" (UID: \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\") " Oct 30 18:03:34.611130 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.611055 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e0d420-e5f8-4eb6-8764-ddd26b284227-kube-api-access-trjnl\" (UniqueName: \"kubernetes.io/projected/e5e0d420-e5f8-4eb6-8764-ddd26b284227-kube-api-access-trjnl\") pod \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\" (UID: \"e5e0d420-e5f8-4eb6-8764-ddd26b284227\") " Oct 30 18:03:34.611239 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.611216 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-openshift-ca" pod "e5e0d420-e5f8-4eb6-8764-ddd26b284227" (UID: "e5e0d420-e5f8-4eb6-8764-ddd26b284227"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:34.611401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.611383 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:03:34.613112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.613077 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e0d420-e5f8-4eb6-8764-ddd26b284227-kube-api-access-trjnl" pod "e5e0d420-e5f8-4eb6-8764-ddd26b284227" (UID: "e5e0d420-e5f8-4eb6-8764-ddd26b284227"). InnerVolumeSpecName "kube-api-access-trjnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:34.613226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.613078 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-xks-ca" pod "e5e0d420-e5f8-4eb6-8764-ddd26b284227" (UID: "e5e0d420-e5f8-4eb6-8764-ddd26b284227"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:34.712326 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.712285 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e0d420-e5f8-4eb6-8764-ddd26b284227-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:03:34.712326 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:34.712318 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trjnl\" (UniqueName: \"kubernetes.io/projected/e5e0d420-e5f8-4eb6-8764-ddd26b284227-kube-api-access-trjnl\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:03:35.079501 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:03:35.079417 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:03:35.302762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:35.302721 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" Oct 30 18:03:35.302762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:35.302752 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm" event={"ID":"e5e0d420-e5f8-4eb6-8764-ddd26b284227","Type":"ContainerDied","Data":"77a370f904e0bcf7e7670bb85a0554eb582054c7bfad0ed7b2e5e7b06e256c8c"} Oct 30 18:03:35.303011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:35.302801 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77a370f904e0bcf7e7670bb85a0554eb582054c7bfad0ed7b2e5e7b06e256c8c" Oct 30 18:03:41.635106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:41.635068 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl"] Oct 30 18:03:41.648302 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:41.648271 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce05c7ef-llqcl"] Oct 30 18:03:42.850461 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:42.850419 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k"] Oct 30 18:03:42.864156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:42.864120 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8c61e05-hd24k"] Oct 30 18:03:43.177333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:43.177240 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca0b20a-3101-4db4-9f65-87d88a8a89c2" path="/var/lib/kubelet/pods/0ca0b20a-3101-4db4-9f65-87d88a8a89c2/volumes" Oct 30 18:03:43.177705 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:43.177688 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8856cb09-5a3c-4d33-80da-21dda441b2a1" path="/var/lib/kubelet/pods/8856cb09-5a3c-4d33-80da-21dda441b2a1/volumes" Oct 30 18:03:45.872859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.872809 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc"] Oct 30 18:03:45.873295 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.873247 4883 cpu_manager.go:414] "containerMap: removing container" podUID="e5e0d420-e5f8-4eb6-8764-ddd26b284227" containerName="revoke-keys" Oct 30 18:03:45.873295 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.873260 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e0d420-e5f8-4eb6-8764-ddd26b284227" containerName="revoke-keys" Oct 30 18:03:45.873295 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.873295 4883 cpu_manager.go:414] "containerMap: removing container" podUID="0ca0b20a-3101-4db4-9f65-87d88a8a89c2" containerName="revoke-keys" Oct 30 18:03:45.873455 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.873301 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca0b20a-3101-4db4-9f65-87d88a8a89c2" containerName="revoke-keys" Oct 30 18:03:45.873455 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.873378 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca0b20a-3101-4db4-9f65-87d88a8a89c2" containerName="revoke-keys" Oct 30 18:03:45.873455 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.873392 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e0d420-e5f8-4eb6-8764-ddd26b284227" containerName="revoke-keys" Oct 30 18:03:45.878525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.878494 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:45.913223 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.913186 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-419d32-openshift-default-dockercfg-l4pzs\"" Oct 30 18:03:45.913413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.913282 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:45.913413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.913320 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s96xh\" (UniqueName: \"kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-kube-api-access-s96xh\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:45.913540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.913445 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:45.913540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.913475 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:45.913540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.913505 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:45.913699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.913598 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:45.913699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.913650 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:45.913801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.913697 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:45.913801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.913730 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:45.915281 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:45.915249 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc"] Oct 30 18:03:46.014518 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.014476 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.014821 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.014778 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.014987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.014860 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.014987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.014887 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s96xh\" (UniqueName: \"kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-kube-api-access-s96xh\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.014987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.014924 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.014987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.014974 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.015217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.015006 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.015217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.015047 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.015217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.015108 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.015695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.015652 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.016180 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.016156 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.016284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.016268 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.016976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.016913 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.017172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.017152 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.018681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.018650 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.028427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.028394 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.050964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.049174 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.052888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.052853 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s96xh\" (UniqueName: \"kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-kube-api-access-s96xh\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.197980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.197871 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:46.565081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.565017 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc"] Oct 30 18:03:46.568585 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:03:46.568552 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7981a18_b6ed_477f_abaa_39ef9490ec6f.slice/crio-d286d7e0b756259723be26ea69ab7d7a22e0efbecea3f1408c9993d980b2b22f WatchSource:0}: Error finding container d286d7e0b756259723be26ea69ab7d7a22e0efbecea3f1408c9993d980b2b22f: Status 404 returned error can't find the container with id d286d7e0b756259723be26ea69ab7d7a22e0efbecea3f1408c9993d980b2b22f Oct 30 18:03:46.571061 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.571023 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:03:46.571148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.571097 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:03:46.571148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:46.571128 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:03:47.196146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:47.196097 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm"] Oct 30 18:03:47.202220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:47.202184 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5e0da928-7xxhm"] Oct 30 18:03:47.366977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:47.366924 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" event={"ID":"b7981a18-b6ed-477f-abaa-39ef9490ec6f","Type":"ContainerStarted","Data":"e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c"} Oct 30 18:03:47.367144 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:47.366997 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" event={"ID":"b7981a18-b6ed-477f-abaa-39ef9490ec6f","Type":"ContainerStarted","Data":"d286d7e0b756259723be26ea69ab7d7a22e0efbecea3f1408c9993d980b2b22f"} Oct 30 18:03:47.436884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:47.436578 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" podStartSLOduration=2.436559681 podStartE2EDuration="2.436559681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:47.434684007 +0000 UTC m=+686.391687843" watchObservedRunningTime="2026-10-30 18:03:47.436559681 +0000 UTC m=+686.393563589" Oct 30 18:03:48.199095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:48.199052 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:48.204227 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:48.204197 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:48.375846 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:48.375814 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:48.376847 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:48.376825 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:03:49.177192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:49.177157 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e0d420-e5f8-4eb6-8764-ddd26b284227" path="/var/lib/kubelet/pods/e5e0d420-e5f8-4eb6-8764-ddd26b284227/volumes" Oct 30 18:03:51.853433 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:51.853398 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl"] Oct 30 18:03:51.858310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:51.858276 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:03:51.880371 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:51.880337 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:03:51.880555 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:51.880436 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:03:51.880555 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:51.880461 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:03:51.894154 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:51.894120 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:03:51.896092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:51.896056 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl"] Oct 30 18:03:51.975741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:51.975706 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2x2\" (UniqueName: \"kubernetes.io/projected/b9a89dec-7d8e-4af5-806b-ef80f369c362-kube-api-access-kt2x2\") pod \"maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl\" (UID: \"b9a89dec-7d8e-4af5-806b-ef80f369c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:03:51.975956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:51.975755 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl\" (UID: \"b9a89dec-7d8e-4af5-806b-ef80f369c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:03:51.975956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:51.975850 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl\" (UID: \"b9a89dec-7d8e-4af5-806b-ef80f369c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:03:52.076388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:52.076346 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2x2\" (UniqueName: \"kubernetes.io/projected/b9a89dec-7d8e-4af5-806b-ef80f369c362-kube-api-access-kt2x2\") pod \"maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl\" (UID: \"b9a89dec-7d8e-4af5-806b-ef80f369c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:03:52.076584 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:52.076403 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl\" (UID: \"b9a89dec-7d8e-4af5-806b-ef80f369c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:03:52.076584 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:52.076441 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl\" (UID: \"b9a89dec-7d8e-4af5-806b-ef80f369c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:03:52.077175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:52.077151 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl\" (UID: \"b9a89dec-7d8e-4af5-806b-ef80f369c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:03:52.078909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:52.078884 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl\" (UID: \"b9a89dec-7d8e-4af5-806b-ef80f369c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:03:52.108881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:52.108796 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2x2\" (UniqueName: \"kubernetes.io/projected/b9a89dec-7d8e-4af5-806b-ef80f369c362-kube-api-access-kt2x2\") pod \"maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl\" (UID: \"b9a89dec-7d8e-4af5-806b-ef80f369c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:03:52.175889 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:52.175852 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:03:52.479986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:52.479865 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl"] Oct 30 18:03:52.483184 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:03:52.483149 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a89dec_7d8e_4af5_806b_ef80f369c362.slice/crio-602612e1f85d4f6b688b2d69898f44805668cb8197273e9cfbb866f3eb8819a3 WatchSource:0}: Error finding container 602612e1f85d4f6b688b2d69898f44805668cb8197273e9cfbb866f3eb8819a3: Status 404 returned error can't find the container with id 602612e1f85d4f6b688b2d69898f44805668cb8197273e9cfbb866f3eb8819a3 Oct 30 18:03:53.405151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:53.405117 4883 generic.go:356] "Generic (PLEG): container finished" podID="b9a89dec-7d8e-4af5-806b-ef80f369c362" containerID="3ff88201be32a0da35d57cfc9906ff8f1a602608803727c111ec6f6518addf29" exitCode=0 Oct 30 18:03:53.405566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:53.405206 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" event={"ID":"b9a89dec-7d8e-4af5-806b-ef80f369c362","Type":"ContainerDied","Data":"3ff88201be32a0da35d57cfc9906ff8f1a602608803727c111ec6f6518addf29"} Oct 30 18:03:53.405566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:53.405267 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" event={"ID":"b9a89dec-7d8e-4af5-806b-ef80f369c362","Type":"ContainerStarted","Data":"602612e1f85d4f6b688b2d69898f44805668cb8197273e9cfbb866f3eb8819a3"} Oct 30 18:03:54.558092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:54.558052 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:03:54.601801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:54.601766 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a89dec-7d8e-4af5-806b-ef80f369c362-kube-api-access-kt2x2\" (UniqueName: \"kubernetes.io/projected/b9a89dec-7d8e-4af5-806b-ef80f369c362-kube-api-access-kt2x2\") pod \"b9a89dec-7d8e-4af5-806b-ef80f369c362\" (UID: \"b9a89dec-7d8e-4af5-806b-ef80f369c362\") " Oct 30 18:03:54.602017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:54.601830 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-openshift-ca\") pod \"b9a89dec-7d8e-4af5-806b-ef80f369c362\" (UID: \"b9a89dec-7d8e-4af5-806b-ef80f369c362\") " Oct 30 18:03:54.602017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:54.601924 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-xks-ca\") pod \"b9a89dec-7d8e-4af5-806b-ef80f369c362\" (UID: \"b9a89dec-7d8e-4af5-806b-ef80f369c362\") " Oct 30 18:03:54.602330 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:54.602297 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-openshift-ca" pod "b9a89dec-7d8e-4af5-806b-ef80f369c362" (UID: "b9a89dec-7d8e-4af5-806b-ef80f369c362"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:54.602443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:54.602392 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:03:54.604425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:54.604396 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-xks-ca" pod "b9a89dec-7d8e-4af5-806b-ef80f369c362" (UID: "b9a89dec-7d8e-4af5-806b-ef80f369c362"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:54.604681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:54.604650 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a89dec-7d8e-4af5-806b-ef80f369c362-kube-api-access-kt2x2" pod "b9a89dec-7d8e-4af5-806b-ef80f369c362" (UID: "b9a89dec-7d8e-4af5-806b-ef80f369c362"). InnerVolumeSpecName "kube-api-access-kt2x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:54.703806 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:54.703718 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt2x2\" (UniqueName: \"kubernetes.io/projected/b9a89dec-7d8e-4af5-806b-ef80f369c362-kube-api-access-kt2x2\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:03:54.703806 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:54.703751 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a89dec-7d8e-4af5-806b-ef80f369c362-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:03:55.420639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:55.420599 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" event={"ID":"b9a89dec-7d8e-4af5-806b-ef80f369c362","Type":"ContainerDied","Data":"602612e1f85d4f6b688b2d69898f44805668cb8197273e9cfbb866f3eb8819a3"} Oct 30 18:03:55.420834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:55.420660 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="602612e1f85d4f6b688b2d69898f44805668cb8197273e9cfbb866f3eb8819a3" Oct 30 18:03:55.420834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:03:55.420721 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl" Oct 30 18:04:05.079499 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:04:05.079413 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:04:08.513471 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:08.513435 4883 generic.go:356] "Generic (PLEG): container finished" podID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerID="e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c" exitCode=0 Oct 30 18:04:08.514043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:08.513517 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" event={"ID":"b7981a18-b6ed-477f-abaa-39ef9490ec6f","Type":"ContainerDied","Data":"e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c"} Oct 30 18:04:08.514137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:08.514068 4883 scope.go:122] "RemoveContainer" containerID="e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c" Oct 30 18:04:08.515415 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:08.515379 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:04:08.515497 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:08.515455 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:04:08.515497 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:08.515489 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:04:09.522189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:09.521468 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" event={"ID":"b7981a18-b6ed-477f-abaa-39ef9490ec6f","Type":"ContainerStarted","Data":"3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7"} Oct 30 18:04:10.198262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:10.198221 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:04:10.198575 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:10.198546 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:04:10.198679 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:10.198617 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 18:04:11.198688 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:11.198655 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:04:11.199142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:11.198719 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 18:04:11.680961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:11.673160 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl"] Oct 30 18:04:11.684956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:11.683127 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9a40825-bb6cl"] Oct 30 18:04:12.198597 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:12.198553 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:04:12.198813 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:12.198635 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 18:04:13.194962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:13.191676 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a89dec-7d8e-4af5-806b-ef80f369c362" path="/var/lib/kubelet/pods/b9a89dec-7d8e-4af5-806b-ef80f369c362/volumes" Oct 30 18:04:13.198824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:13.198781 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:04:13.199269 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:13.198852 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 18:04:14.199031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:14.198658 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 18:04:14.199031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:14.198707 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 18:04:15.203589 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.203557 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:04:15.204062 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.203845 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:04:15.204733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.204713 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:04:15.549051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.548964 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl"] Oct 30 18:04:15.549574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.549550 4883 cpu_manager.go:414] "containerMap: removing container" podUID="b9a89dec-7d8e-4af5-806b-ef80f369c362" containerName="revoke-keys" Oct 30 18:04:15.549574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.549570 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a89dec-7d8e-4af5-806b-ef80f369c362" containerName="revoke-keys" Oct 30 18:04:15.549761 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.549674 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a89dec-7d8e-4af5-806b-ef80f369c362" containerName="revoke-keys" Oct 30 18:04:15.554703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.553529 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.573115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.573057 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6e8fca\"/\"openshift-service-ca.crt\"" Oct 30 18:04:15.573354 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.573128 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6e8fca\"/\"kube-root-ca.crt\"" Oct 30 18:04:15.573354 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.573281 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6e8fca\"/\"default-dockercfg-d6n2t\"" Oct 30 18:04:15.573354 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.573313 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6e8fca\"/\"simulator-w2-6e8fca-kserve-self-signed-certs\"" Oct 30 18:04:15.575520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.575496 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl"] Oct 30 18:04:15.655095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.655055 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16a36bd-5d13-496e-bbef-965ca65aec24-tls-certs\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.655283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.655158 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-dshm\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.655399 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.655379 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-home\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.655448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.655431 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-tmp-dir\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.655489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.655469 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfvmp\" (UniqueName: \"kubernetes.io/projected/f16a36bd-5d13-496e-bbef-965ca65aec24-kube-api-access-qfvmp\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.655544 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.655503 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-model-cache\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.756162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.756119 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-dshm\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.756359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.756209 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-home\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.756359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.756243 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-tmp-dir\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.756359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.756267 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfvmp\" (UniqueName: \"kubernetes.io/projected/f16a36bd-5d13-496e-bbef-965ca65aec24-kube-api-access-qfvmp\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.756359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.756300 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-model-cache\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.756359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.756350 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16a36bd-5d13-496e-bbef-965ca65aec24-tls-certs\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.756689 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.756658 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-home\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.756771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.756685 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-tmp-dir\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.756771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.756739 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-model-cache\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.758513 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.758492 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-dshm\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.758771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.758754 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16a36bd-5d13-496e-bbef-965ca65aec24-tls-certs\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.781250 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.781217 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfvmp\" (UniqueName: \"kubernetes.io/projected/f16a36bd-5d13-496e-bbef-965ca65aec24-kube-api-access-qfvmp\") pod \"simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:15.868205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:15.868161 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:16.174193 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:16.174111 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl"] Oct 30 18:04:16.179408 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:04:16.179379 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf16a36bd_5d13_496e_bbef_965ca65aec24.slice/crio-e827cde89f0e3806920640cae1caecd366111e3a08ccd2afe5f500b28361b3c9 WatchSource:0}: Error finding container e827cde89f0e3806920640cae1caecd366111e3a08ccd2afe5f500b28361b3c9: Status 404 returned error can't find the container with id e827cde89f0e3806920640cae1caecd366111e3a08ccd2afe5f500b28361b3c9 Oct 30 18:04:16.562759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:16.562666 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" event={"ID":"f16a36bd-5d13-496e-bbef-965ca65aec24","Type":"ContainerStarted","Data":"e827cde89f0e3806920640cae1caecd366111e3a08ccd2afe5f500b28361b3c9"} Oct 30 18:04:17.568984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:17.568917 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" event={"ID":"f16a36bd-5d13-496e-bbef-965ca65aec24","Type":"ContainerStarted","Data":"44b9673f6528e93cd3b0176a1e8c49e207f2dcb3ff7cc36119476177c8d59b15"} Oct 30 18:04:17.614352 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:17.614218 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podStartSLOduration=1.449470303 podStartE2EDuration="2.614203813s" totalImagesPullingTime="1.16473351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:17.613169546 +0000 UTC m=+716.570173382" watchObservedRunningTime="2026-10-30 18:04:17.614203813 +0000 UTC m=+716.571207643" Oct 30 18:04:17.868456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:17.868370 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:25.869279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:25.869237 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:25.882096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:25.882066 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:26.631012 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:26.630972 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:04:27.112288 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.112252 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns"] Oct 30 18:04:27.117369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.117343 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.145426 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.145390 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns"] Oct 30 18:04:27.153696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.153657 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-419d32\"/\"default-dockercfg-5z8s9\"" Oct 30 18:04:27.153696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.153662 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-419d32\"/\"simulator-w4-419d32-kserve-self-signed-certs\"" Oct 30 18:04:27.172834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.172798 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-tmp-dir\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.172834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.172837 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-home\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.173080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.172858 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmvb\" (UniqueName: \"kubernetes.io/projected/eff20d9a-c474-4b76-809d-add74ef702af-kube-api-access-5vmvb\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.173080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.173061 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-dshm\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.173186 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.173169 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff20d9a-c474-4b76-809d-add74ef702af-tls-certs\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.173250 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.173207 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-model-cache\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.203824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.203785 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-419d32\"/\"kube-root-ca.crt\"" Oct 30 18:04:27.204005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.203825 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-419d32\"/\"openshift-service-ca.crt\"" Oct 30 18:04:27.274622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.274583 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff20d9a-c474-4b76-809d-add74ef702af-tls-certs\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.274622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.274637 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-model-cache\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.274878 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.274696 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-tmp-dir\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.274878 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.274721 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-home\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.274878 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.274750 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmvb\" (UniqueName: \"kubernetes.io/projected/eff20d9a-c474-4b76-809d-add74ef702af-kube-api-access-5vmvb\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.274878 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.274819 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-dshm\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.275254 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.275229 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-model-cache\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.275522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.275494 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-tmp-dir\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.275630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.275609 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-home\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.277752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.277719 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff20d9a-c474-4b76-809d-add74ef702af-tls-certs\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.277874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.277786 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-dshm\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.303361 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.303324 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmvb\" (UniqueName: \"kubernetes.io/projected/eff20d9a-c474-4b76-809d-add74ef702af-kube-api-access-5vmvb\") pod \"simulator-w4-419d32-kserve-8dc97d64b-r6zns\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.430738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.430640 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:27.742066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:27.741981 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns"] Oct 30 18:04:27.745449 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:04:27.745419 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeff20d9a_c474_4b76_809d_add74ef702af.slice/crio-0a8ed2bf1fd651cb7449144a1760b708e4330dbf8ed0fc5332de373b8fa13149 WatchSource:0}: Error finding container 0a8ed2bf1fd651cb7449144a1760b708e4330dbf8ed0fc5332de373b8fa13149: Status 404 returned error can't find the container with id 0a8ed2bf1fd651cb7449144a1760b708e4330dbf8ed0fc5332de373b8fa13149 Oct 30 18:04:28.630490 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:28.630454 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" event={"ID":"eff20d9a-c474-4b76-809d-add74ef702af","Type":"ContainerStarted","Data":"6ae38f383fed3d0da9688e8e00dafe14ad6625f7ed23e538885672d4ce1808f7"} Oct 30 18:04:28.630490 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:28.630495 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" event={"ID":"eff20d9a-c474-4b76-809d-add74ef702af","Type":"ContainerStarted","Data":"0a8ed2bf1fd651cb7449144a1760b708e4330dbf8ed0fc5332de373b8fa13149"} Oct 30 18:04:28.682653 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:28.682592 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podStartSLOduration=1.682577043 podStartE2EDuration="1.682577043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:28.679988911 +0000 UTC m=+727.636992748" watchObservedRunningTime="2026-10-30 18:04:28.682577043 +0000 UTC m=+727.639580878" Oct 30 18:04:29.430998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:29.430929 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:35.080732 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:04:35.080688 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:04:37.431950 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:37.431836 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:37.451697 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:37.451666 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:37.692570 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:37.692467 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:04:39.218698 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.218662 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj"] Oct 30 18:04:39.222441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.222411 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.251731 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.251694 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3cc458\"/\"default-dockercfg-tfdlb\"" Oct 30 18:04:39.251920 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.251862 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3cc458\"/\"openshift-service-ca.crt\"" Oct 30 18:04:39.251920 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.251874 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3cc458\"/\"simulator-w3-3cc458-kserve-self-signed-certs\"" Oct 30 18:04:39.252094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.252079 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3cc458\"/\"kube-root-ca.crt\"" Oct 30 18:04:39.254401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.254379 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj"] Oct 30 18:04:39.315127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.315089 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-tmp-dir\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.315285 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.315223 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz7wl\" (UniqueName: \"kubernetes.io/projected/cf31407f-9393-4e29-abd5-11332bf68e81-kube-api-access-zz7wl\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.315285 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.315277 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-dshm\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.315393 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.315327 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-model-cache\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.315468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.315445 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf31407f-9393-4e29-abd5-11332bf68e81-tls-certs\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.315523 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.315506 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-home\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.416240 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.416199 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf31407f-9393-4e29-abd5-11332bf68e81-tls-certs\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.416240 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.416248 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-home\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.416483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.416279 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-tmp-dir\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.416483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.416318 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz7wl\" (UniqueName: \"kubernetes.io/projected/cf31407f-9393-4e29-abd5-11332bf68e81-kube-api-access-zz7wl\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.416595 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.416477 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-dshm\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.416595 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.416543 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-model-cache\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.416765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.416746 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-home\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.416829 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.416787 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-tmp-dir\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.416889 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.416874 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-model-cache\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.418765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.418745 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-dshm\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.418948 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.418917 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf31407f-9393-4e29-abd5-11332bf68e81-tls-certs\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.453981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.453926 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz7wl\" (UniqueName: \"kubernetes.io/projected/cf31407f-9393-4e29-abd5-11332bf68e81-kube-api-access-zz7wl\") pod \"simulator-w3-3cc458-kserve-b9598ddbb-4znlj\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.537669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.537589 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:39.850829 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:39.850722 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj"] Oct 30 18:04:39.864215 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:04:39.864174 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf31407f_9393_4e29_abd5_11332bf68e81.slice/crio-6a3f55ae163078f0925afe27aecddba58def7961aae3a7b9e35aa809c781aaad WatchSource:0}: Error finding container 6a3f55ae163078f0925afe27aecddba58def7961aae3a7b9e35aa809c781aaad: Status 404 returned error can't find the container with id 6a3f55ae163078f0925afe27aecddba58def7961aae3a7b9e35aa809c781aaad Oct 30 18:04:40.699546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:40.699505 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" event={"ID":"cf31407f-9393-4e29-abd5-11332bf68e81","Type":"ContainerStarted","Data":"dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43"} Oct 30 18:04:40.699991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:40.699563 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" event={"ID":"cf31407f-9393-4e29-abd5-11332bf68e81","Type":"ContainerStarted","Data":"6a3f55ae163078f0925afe27aecddba58def7961aae3a7b9e35aa809c781aaad"} Oct 30 18:04:40.745123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:40.745053 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podStartSLOduration=1.745029128 podStartE2EDuration="1.745029128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:40.743277787 +0000 UTC m=+739.700281611" watchObservedRunningTime="2026-10-30 18:04:40.745029128 +0000 UTC m=+739.702032967" Oct 30 18:04:41.538169 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:41.538130 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:43.301481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.301444 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r"] Oct 30 18:04:43.306237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.306206 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.339496 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.339456 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6e8fca-openshift-default-dockercfg-sjtwn\"" Oct 30 18:04:43.343696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.343659 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r"] Oct 30 18:04:43.458463 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.458395 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0086e9f0-1201-4263-b710-5417225349ee-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.458463 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.458468 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.458706 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.458542 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.458706 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.458564 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.458706 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.458608 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.458818 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.458738 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.458818 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.458760 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0086e9f0-1201-4263-b710-5417225349ee-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.458818 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.458803 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.458915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.458846 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpdhs\" (UniqueName: \"kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-kube-api-access-vpdhs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.559752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.559657 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0086e9f0-1201-4263-b710-5417225349ee-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.559752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.559714 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.559752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.559742 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.560080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.559766 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.560080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.559830 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.560080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.559870 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.560080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.559897 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0086e9f0-1201-4263-b710-5417225349ee-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.560080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.559981 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.560080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.560015 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpdhs\" (UniqueName: \"kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-kube-api-access-vpdhs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.560371 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.560265 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.560463 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.560438 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.560589 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.560570 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0086e9f0-1201-4263-b710-5417225349ee-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.560643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.560623 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.560730 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.560710 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.562505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.562480 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.562813 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.562792 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0086e9f0-1201-4263-b710-5417225349ee-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.586620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.586587 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.586775 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.586743 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpdhs\" (UniqueName: \"kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-kube-api-access-vpdhs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.622150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.622111 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:43.945271 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.945143 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r"] Oct 30 18:04:43.948275 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:04:43.948245 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0086e9f0_1201_4263_b710_5417225349ee.slice/crio-84a1229ab50f804fc95120f3cbaff303a88992bab900e1ee9393dd718c8655af WatchSource:0}: Error finding container 84a1229ab50f804fc95120f3cbaff303a88992bab900e1ee9393dd718c8655af: Status 404 returned error can't find the container with id 84a1229ab50f804fc95120f3cbaff303a88992bab900e1ee9393dd718c8655af Oct 30 18:04:43.950854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.950812 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:04:43.950988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.950904 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:04:43.950988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:43.950968 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:04:44.723545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:44.723504 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" event={"ID":"0086e9f0-1201-4263-b710-5417225349ee","Type":"ContainerStarted","Data":"9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568"} Oct 30 18:04:44.724090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:44.723606 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" event={"ID":"0086e9f0-1201-4263-b710-5417225349ee","Type":"ContainerStarted","Data":"84a1229ab50f804fc95120f3cbaff303a88992bab900e1ee9393dd718c8655af"} Oct 30 18:04:44.794252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:44.793572 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" podStartSLOduration=1.7935500370000002 podStartE2EDuration="1.793550037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:44.790645836 +0000 UTC m=+743.747649673" watchObservedRunningTime="2026-10-30 18:04:44.793550037 +0000 UTC m=+743.750553874" Oct 30 18:04:45.623397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:45.623350 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:45.625271 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:45.625237 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:04:45.625381 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:45.625304 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" podUID="0086e9f0-1201-4263-b710-5417225349ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 18:04:46.623171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:46.623130 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:04:46.623673 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:46.623183 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" podUID="0086e9f0-1201-4263-b710-5417225349ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 18:04:47.623103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:47.623065 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:04:47.623328 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:47.623132 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" podUID="0086e9f0-1201-4263-b710-5417225349ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 18:04:48.622953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:48.622907 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:04:48.623145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:48.622990 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" podUID="0086e9f0-1201-4263-b710-5417225349ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 18:04:49.537803 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:49.537767 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:49.550435 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:49.550399 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:49.622500 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:49.622456 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:04:49.622702 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:49.622521 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" podUID="0086e9f0-1201-4263-b710-5417225349ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 18:04:49.762056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:49.762021 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:04:50.878962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:50.878157 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:50.879454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:50.879409 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:50.880558 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:50.880537 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:04:54.255774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.255738 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp"] Oct 30 18:04:54.262631 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.262600 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.296521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.296478 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6e8fca\"/\"premium-w2-6e8fca-kserve-self-signed-certs\"" Oct 30 18:04:54.298652 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.298618 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp"] Oct 30 18:04:54.302273 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.302234 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-tmp-dir\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.302440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.302287 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-home\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.302440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.302311 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxfj\" (UniqueName: \"kubernetes.io/projected/d8eca37e-ab66-4007-b362-00c0359a2c07-kube-api-access-zkxfj\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.302528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.302510 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-dshm\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.302582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.302562 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-model-cache\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.302701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.302682 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8eca37e-ab66-4007-b362-00c0359a2c07-tls-certs\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.404046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.403989 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-dshm\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.404260 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.404059 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-model-cache\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.404260 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.404137 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8eca37e-ab66-4007-b362-00c0359a2c07-tls-certs\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.404260 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.404188 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-tmp-dir\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.404260 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.404226 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-home\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.404481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.404275 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxfj\" (UniqueName: \"kubernetes.io/projected/d8eca37e-ab66-4007-b362-00c0359a2c07-kube-api-access-zkxfj\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.404565 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.404524 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-model-cache\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.404681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.404660 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-tmp-dir\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.404753 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.404720 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-home\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.407238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.407206 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-dshm\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.407398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.407311 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8eca37e-ab66-4007-b362-00c0359a2c07-tls-certs\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.432117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.432079 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxfj\" (UniqueName: \"kubernetes.io/projected/d8eca37e-ab66-4007-b362-00c0359a2c07-kube-api-access-zkxfj\") pod \"premium-w2-6e8fca-kserve-7cd794c749-dzwbp\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.577205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.577113 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:04:54.885525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:54.885499 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp"] Oct 30 18:04:54.888119 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:04:54.888079 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8eca37e_ab66_4007_b362_00c0359a2c07.slice/crio-1d0bba59153943406eebc202eef1d1ea2dfe30b66808b20c2d8e50d773286194 WatchSource:0}: Error finding container 1d0bba59153943406eebc202eef1d1ea2dfe30b66808b20c2d8e50d773286194: Status 404 returned error can't find the container with id 1d0bba59153943406eebc202eef1d1ea2dfe30b66808b20c2d8e50d773286194 Oct 30 18:04:55.789593 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:55.789553 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" event={"ID":"d8eca37e-ab66-4007-b362-00c0359a2c07","Type":"ContainerStarted","Data":"ad2c56d6ae3710a4bc8370d34269391b549cc60123cf09e81eafb0bfc6e47b21"} Oct 30 18:04:55.789593 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:55.789606 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" event={"ID":"d8eca37e-ab66-4007-b362-00c0359a2c07","Type":"ContainerStarted","Data":"1d0bba59153943406eebc202eef1d1ea2dfe30b66808b20c2d8e50d773286194"} Oct 30 18:04:55.838076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:55.838013 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podStartSLOduration=1.837992915 podStartE2EDuration="1.837992915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:55.835280307 +0000 UTC m=+754.792284143" watchObservedRunningTime="2026-10-30 18:04:55.837992915 +0000 UTC m=+754.794996752" Oct 30 18:04:56.577671 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:04:56.577629 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:05:04.578225 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:04.578124 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:05:04.591109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:04.591078 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:05:04.858670 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:04.858639 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:05:05.081103 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:05:05.081060 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:05:09.196518 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.196472 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp"] Oct 30 18:05:09.200373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.200349 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.231729 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.231693 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-57ee15\"/\"openshift-service-ca.crt\"" Oct 30 18:05:09.231996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.231762 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-57ee15\"/\"default-dockercfg-d54vq\"" Oct 30 18:05:09.231996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.231833 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-57ee15\"/\"kube-root-ca.crt\"" Oct 30 18:05:09.231996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.231873 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-57ee15\"/\"simulator-w0-57ee15-kserve-self-signed-certs\"" Oct 30 18:05:09.234070 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.234049 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp"] Oct 30 18:05:09.283090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.283051 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-model-cache\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.283273 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.283112 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-dshm\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.283273 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.283251 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s69hn\" (UniqueName: \"kubernetes.io/projected/5d215fd2-c798-4224-bff8-4d19fa58b773-kube-api-access-s69hn\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.283389 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.283291 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-tmp-dir\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.283498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.283470 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d215fd2-c798-4224-bff8-4d19fa58b773-tls-certs\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.283737 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.283715 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-home\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.384412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.384369 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d215fd2-c798-4224-bff8-4d19fa58b773-tls-certs\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.384616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.384440 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-home\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.384616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.384476 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-model-cache\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.384616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.384513 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-dshm\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.384616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.384564 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s69hn\" (UniqueName: \"kubernetes.io/projected/5d215fd2-c798-4224-bff8-4d19fa58b773-kube-api-access-s69hn\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.384616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.384602 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-tmp-dir\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.385090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.385060 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-model-cache\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.385192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.385092 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-home\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.385192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.385135 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-tmp-dir\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.387067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.387039 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-dshm\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.388343 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.388322 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d215fd2-c798-4224-bff8-4d19fa58b773-tls-certs\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.415503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.415471 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s69hn\" (UniqueName: \"kubernetes.io/projected/5d215fd2-c798-4224-bff8-4d19fa58b773-kube-api-access-s69hn\") pod \"simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.514970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.514826 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:09.827868 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.827833 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp"] Oct 30 18:05:09.830818 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:05:09.830782 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d215fd2_c798_4224_bff8_4d19fa58b773.slice/crio-e0aae7c9567e2dc88499359da96b150f0591b7f3a18f9df5a3755b43f54aad99 WatchSource:0}: Error finding container e0aae7c9567e2dc88499359da96b150f0591b7f3a18f9df5a3755b43f54aad99: Status 404 returned error can't find the container with id e0aae7c9567e2dc88499359da96b150f0591b7f3a18f9df5a3755b43f54aad99 Oct 30 18:05:09.875709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:09.875677 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" event={"ID":"5d215fd2-c798-4224-bff8-4d19fa58b773","Type":"ContainerStarted","Data":"e0aae7c9567e2dc88499359da96b150f0591b7f3a18f9df5a3755b43f54aad99"} Oct 30 18:05:10.883113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:10.883075 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" event={"ID":"5d215fd2-c798-4224-bff8-4d19fa58b773","Type":"ContainerStarted","Data":"3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef"} Oct 30 18:05:10.930654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:10.930595 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podStartSLOduration=1.930580298 podStartE2EDuration="1.930580298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:10.92870869 +0000 UTC m=+769.885712527" watchObservedRunningTime="2026-10-30 18:05:10.930580298 +0000 UTC m=+769.887584180" Oct 30 18:05:11.515021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:11.514984 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:19.049982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.049922 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5"] Oct 30 18:05:19.055376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.055344 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.072828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.072797 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b424a5-openshift-default-dockercfg-gfhfl\"" Oct 30 18:05:19.087993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.087951 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.088200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.088064 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.088200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.088126 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.088200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.088166 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59db4be-b972-48bc-b2dc-298d5da131dc-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.088200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.088195 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.088421 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.088236 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.088421 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.088292 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.088421 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.088360 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.088421 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.088406 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbqdp\" (UniqueName: \"kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-kube-api-access-nbqdp\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.089183 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.089160 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5"] Oct 30 18:05:19.190030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.189979 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.190246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.190052 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbqdp\" (UniqueName: \"kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-kube-api-access-nbqdp\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.190246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.190107 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.190246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.190150 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.190246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.190195 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.190467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.190251 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59db4be-b972-48bc-b2dc-298d5da131dc-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.190467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.190299 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.190467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.190327 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.190467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.190370 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.190698 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.190670 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.190768 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.190707 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.190826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.190792 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.190826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.190806 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.191088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.191068 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59db4be-b972-48bc-b2dc-298d5da131dc-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.192781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.192761 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.192960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.192926 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.215193 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.215159 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.215467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.215441 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbqdp\" (UniqueName: \"kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-kube-api-access-nbqdp\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.370102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.370061 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:19.515514 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.515471 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:19.529019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.528978 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:19.702475 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.702444 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5"] Oct 30 18:05:19.704337 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:05:19.704309 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59db4be_b972_48bc_b2dc_298d5da131dc.slice/crio-29049ad536236bb1efbb394340c6e7963e6331bdc99e337f0d7bd53391c92c70 WatchSource:0}: Error finding container 29049ad536236bb1efbb394340c6e7963e6331bdc99e337f0d7bd53391c92c70: Status 404 returned error can't find the container with id 29049ad536236bb1efbb394340c6e7963e6331bdc99e337f0d7bd53391c92c70 Oct 30 18:05:19.706657 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.706624 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:05:19.706746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.706690 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:05:19.706746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.706718 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:05:19.951674 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.951571 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" event={"ID":"d59db4be-b972-48bc-b2dc-298d5da131dc","Type":"ContainerStarted","Data":"74f73e3788afb15f9fb12fb3b198e70a6a8b6d103eaf4a2637efe694decbd051"} Oct 30 18:05:19.951674 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.951642 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" event={"ID":"d59db4be-b972-48bc-b2dc-298d5da131dc","Type":"ContainerStarted","Data":"29049ad536236bb1efbb394340c6e7963e6331bdc99e337f0d7bd53391c92c70"} Oct 30 18:05:19.969056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:19.969023 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:05:20.004395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:20.004280 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podStartSLOduration=1.004258626 podStartE2EDuration="1.004258626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:20.001256272 +0000 UTC m=+778.958260109" watchObservedRunningTime="2026-10-30 18:05:20.004258626 +0000 UTC m=+778.961262494" Oct 30 18:05:20.370989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:20.370955 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:20.372656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:20.372628 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:20.372783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:20.372679 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 18:05:21.389416 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:21.389372 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:21.389826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:21.389449 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 18:05:22.382122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:22.382073 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:22.382423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:22.382386 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 18:05:23.373387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:23.373345 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:23.373999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:23.373963 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 18:05:24.376394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:24.376352 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:24.377310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:24.377275 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 18:05:25.319059 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.319014 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv"] Oct 30 18:05:25.327008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.324660 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.352852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.352804 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-home\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.353095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.352905 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b424a5\"/\"kube-root-ca.crt\"" Oct 30 18:05:25.353095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.352930 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lfxp\" (UniqueName: \"kubernetes.io/projected/1ff654c6-de28-43a9-a216-6dc69a9a6752-kube-api-access-7lfxp\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.353095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.353046 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-model-cache\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.353298 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.353134 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-tmp-dir\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.353298 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.353248 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-dshm\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.353298 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.353289 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff654c6-de28-43a9-a216-6dc69a9a6752-tls-certs\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.362926 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.362889 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b424a5\"/\"openshift-service-ca.crt\"" Oct 30 18:05:25.374961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.374131 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv"] Oct 30 18:05:25.375394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.375368 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b424a5\"/\"premium-w1-b424a5-kserve-self-signed-certs\"" Oct 30 18:05:25.375642 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.375626 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b424a5\"/\"default-dockercfg-4tgq7\"" Oct 30 18:05:25.376175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.376152 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:25.376379 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.376352 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 18:05:25.455068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.455024 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-dshm\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.455744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.455094 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff654c6-de28-43a9-a216-6dc69a9a6752-tls-certs\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.455744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.455151 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-home\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.455744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.455182 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lfxp\" (UniqueName: \"kubernetes.io/projected/1ff654c6-de28-43a9-a216-6dc69a9a6752-kube-api-access-7lfxp\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.455744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.455234 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-model-cache\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.455744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.455310 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-tmp-dir\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.456332 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.455804 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-tmp-dir\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.456476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.456451 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-home\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.457219 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.457141 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-model-cache\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.462985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.461545 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-dshm\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.462985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.462869 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff654c6-de28-43a9-a216-6dc69a9a6752-tls-certs\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.517962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.517634 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lfxp\" (UniqueName: \"kubernetes.io/projected/1ff654c6-de28-43a9-a216-6dc69a9a6752-kube-api-access-7lfxp\") pod \"premium-w1-b424a5-kserve-68c9d8696c-nlbqv\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.641168 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.641127 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:25.995217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:25.995182 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv"] Oct 30 18:05:25.995701 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:05:25.995669 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ff654c6_de28_43a9_a216_6dc69a9a6752.slice/crio-f6f07cfa27e44a53b83b04bc53debe52da6ab39b39108af1e50eefe181e2a5e4 WatchSource:0}: Error finding container f6f07cfa27e44a53b83b04bc53debe52da6ab39b39108af1e50eefe181e2a5e4: Status 404 returned error can't find the container with id f6f07cfa27e44a53b83b04bc53debe52da6ab39b39108af1e50eefe181e2a5e4 Oct 30 18:05:26.015116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:26.014688 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" event={"ID":"1ff654c6-de28-43a9-a216-6dc69a9a6752","Type":"ContainerStarted","Data":"f6f07cfa27e44a53b83b04bc53debe52da6ab39b39108af1e50eefe181e2a5e4"} Oct 30 18:05:26.370885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:26.370840 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:05:26.371112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:26.370912 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 18:05:27.022309 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:27.022263 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" event={"ID":"1ff654c6-de28-43a9-a216-6dc69a9a6752","Type":"ContainerStarted","Data":"9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc"} Oct 30 18:05:27.117253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:27.117171 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podStartSLOduration=2.117148427 podStartE2EDuration="2.117148427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:27.11338748 +0000 UTC m=+786.070391316" watchObservedRunningTime="2026-10-30 18:05:27.117148427 +0000 UTC m=+786.074152265" Oct 30 18:05:27.376854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:27.376822 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:27.377137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:27.377119 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:27.377828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:27.377812 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:05:27.641719 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:27.641625 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:28.505956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.505265 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj"] Oct 30 18:05:28.513964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.510816 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.548430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.548389 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj"] Oct 30 18:05:28.605956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.605445 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.605956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.605513 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrb6k\" (UniqueName: \"kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-kube-api-access-qrb6k\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.605956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.605623 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b7c407-135e-45ee-96b6-b09469351896-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.605956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.605682 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b7c407-135e-45ee-96b6-b09469351896-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.605956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.605754 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.605956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.605782 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.605956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.605908 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.606468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.605986 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.606468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.606049 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.706998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.706958 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.707196 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.707012 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.707196 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.707049 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.707196 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.707114 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.707359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.707139 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrb6k\" (UniqueName: \"kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-kube-api-access-qrb6k\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.707541 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.707519 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b7c407-135e-45ee-96b6-b09469351896-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.707736 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.707596 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.707814 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.707692 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b7c407-135e-45ee-96b6-b09469351896-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.707870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.707824 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.707870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.707856 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.708160 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.708133 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.708238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.708216 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.708289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.708277 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.708343 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.708316 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b7c407-135e-45ee-96b6-b09469351896-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.710713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.710689 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.710790 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.710722 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b7c407-135e-45ee-96b6-b09469351896-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.732628 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.732587 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrb6k\" (UniqueName: \"kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-kube-api-access-qrb6k\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.733671 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.733642 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:28.837539 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:28.837450 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:29.198182 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:05:29.198103 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b7c407_135e_45ee_96b6_b09469351896.slice/crio-39ae3617f251e6ea1f76f1054ce9d085f2380c8e4d276b7745753d4271d7c804 WatchSource:0}: Error finding container 39ae3617f251e6ea1f76f1054ce9d085f2380c8e4d276b7745753d4271d7c804: Status 404 returned error can't find the container with id 39ae3617f251e6ea1f76f1054ce9d085f2380c8e4d276b7745753d4271d7c804 Oct 30 18:05:29.200426 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:29.200380 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:05:29.200541 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:29.200478 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:05:29.200541 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:29.200521 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:05:29.201883 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:29.201853 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj"] Oct 30 18:05:30.042530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:30.042492 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" event={"ID":"30b7c407-135e-45ee-96b6-b09469351896","Type":"ContainerStarted","Data":"c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c"} Oct 30 18:05:30.042924 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:30.042581 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" event={"ID":"30b7c407-135e-45ee-96b6-b09469351896","Type":"ContainerStarted","Data":"39ae3617f251e6ea1f76f1054ce9d085f2380c8e4d276b7745753d4271d7c804"} Oct 30 18:05:30.110949 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:30.110886 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" podStartSLOduration=2.110870125 podStartE2EDuration="2.110870125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:30.108297797 +0000 UTC m=+789.065301632" watchObservedRunningTime="2026-10-30 18:05:30.110870125 +0000 UTC m=+789.067873961" Oct 30 18:05:30.837987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:30.837919 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:30.839643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:30.839613 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:05:30.839755 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:30.839666 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:05:31.838703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:31.838669 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:05:31.839100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:31.838726 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:05:32.838397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:32.838356 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:05:32.838590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:32.838419 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:05:33.838474 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:33.838435 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:05:33.838899 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:33.838499 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:05:34.838152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:34.838090 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:05:34.838369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:34.838168 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:05:35.105185 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:05:35.105143 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:05:35.641351 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:35.641313 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:35.654056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:35.654018 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:35.838672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:35.838635 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:05:35.838852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:35.838686 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 18:05:36.110146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:36.110108 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:05:36.854131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:36.854095 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:36.855043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:36.854920 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:36.855779 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:36.855759 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:05:39.006151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.006102 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr"] Oct 30 18:05:39.010377 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.010351 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.051386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.051351 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr"] Oct 30 18:05:39.093452 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.093415 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-57ee15-openshift-default-dockercfg-jlxgq\"" Oct 30 18:05:39.155464 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.155413 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.155658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.155485 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z2d5\" (UniqueName: \"kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-kube-api-access-8z2d5\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.155658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.155590 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.155767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.155662 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f847895f-c06c-4e79-8f10-9305e980856a-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.155767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.155706 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.155875 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.155821 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.155875 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.155866 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.156036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.156016 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.156103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.156087 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f847895f-c06c-4e79-8f10-9305e980856a-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.257690 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.257590 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f847895f-c06c-4e79-8f10-9305e980856a-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.257974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.257925 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.258105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.257998 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.258105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.258025 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.258105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.258068 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.258259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.258122 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f847895f-c06c-4e79-8f10-9305e980856a-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.258259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.258196 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.258259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.258231 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z2d5\" (UniqueName: \"kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-kube-api-access-8z2d5\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.258259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.258248 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.258541 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.258516 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.258801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.258779 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.259241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.259222 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.259846 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.259356 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f847895f-c06c-4e79-8f10-9305e980856a-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.260036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.259535 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.260865 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.260842 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f847895f-c06c-4e79-8f10-9305e980856a-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.261672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.261654 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.285717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.285684 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.285884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.285859 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z2d5\" (UniqueName: \"kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-kube-api-access-8z2d5\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.325148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.325112 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:39.652861 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.652827 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr"] Oct 30 18:05:39.655598 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:05:39.655570 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf847895f_c06c_4e79_8f10_9305e980856a.slice/crio-e64c286edb1bc49ff69c8b8f51ae9ce7580991f06e648908eff332d9f9b70644 WatchSource:0}: Error finding container e64c286edb1bc49ff69c8b8f51ae9ce7580991f06e648908eff332d9f9b70644: Status 404 returned error can't find the container with id e64c286edb1bc49ff69c8b8f51ae9ce7580991f06e648908eff332d9f9b70644 Oct 30 18:05:39.658115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.658077 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:05:39.658229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.658167 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:05:39.658229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:39.658215 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:05:40.123979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:40.123928 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" event={"ID":"f847895f-c06c-4e79-8f10-9305e980856a","Type":"ContainerStarted","Data":"e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc"} Oct 30 18:05:40.124427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:40.124002 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" event={"ID":"f847895f-c06c-4e79-8f10-9305e980856a","Type":"ContainerStarted","Data":"e64c286edb1bc49ff69c8b8f51ae9ce7580991f06e648908eff332d9f9b70644"} Oct 30 18:05:40.177807 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:40.177746 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" podStartSLOduration=2.177728943 podStartE2EDuration="2.177728943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:40.176052698 +0000 UTC m=+799.133056538" watchObservedRunningTime="2026-10-30 18:05:40.177728943 +0000 UTC m=+799.134732778" Oct 30 18:05:40.325758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:40.325718 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:40.327616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:40.327592 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:05:40.327714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:40.327647 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 18:05:41.326340 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:41.326296 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:05:41.326724 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:41.326354 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 18:05:42.326804 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:42.326765 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:05:42.327346 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:42.326826 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 18:05:43.326069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:43.326026 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:05:43.326261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:43.326087 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 18:05:44.325546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:44.325501 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:05:44.326107 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:44.325567 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 18:05:45.326260 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:45.326222 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:05:45.326866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:45.326831 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 18:05:46.455447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:46.455413 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:46.455881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:46.455737 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:46.456492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:46.456470 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:05:54.197096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.197056 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm"] Oct 30 18:05:54.202470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.202439 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.221186 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.221154 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3cc458-openshift-default-dockercfg-ld6nz\"" Oct 30 18:05:54.238691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.238648 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm"] Oct 30 18:05:54.291713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.291678 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.291926 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.291830 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.291926 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.291861 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.291926 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.291913 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.292137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.292102 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.292185 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.292159 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.292238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.292192 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f448\" (UniqueName: \"kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-kube-api-access-8f448\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.292238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.292226 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.292364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.292342 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.393362 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.393325 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f448\" (UniqueName: \"kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-kube-api-access-8f448\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.393559 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.393381 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.393559 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.393430 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.393559 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.393464 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.393737 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.393561 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.393737 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.393587 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.393737 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.393641 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.393737 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.393679 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.393737 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.393722 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.394017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.393829 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.394017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.393920 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.394250 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.394222 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.394329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.394288 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.394424 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.394399 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.396250 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.396219 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.396355 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.396281 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.420042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.420004 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.420192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.420170 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f448\" (UniqueName: \"kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-kube-api-access-8f448\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.519196 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.519097 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:54.775672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.775581 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz"] Oct 30 18:05:54.781050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.781019 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:54.798301 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.798273 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:05:54.800681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.800654 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:05:54.801117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.801093 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:05:54.801488 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.801471 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:05:54.815803 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.815765 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz"] Oct 30 18:05:54.849987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.849958 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm"] Oct 30 18:05:54.852648 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:05:54.852616 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b5a129_7409_43f5_8ffc_5c6b7738cdb4.slice/crio-88572bdeb772a472aa0bee6906aadfb956914d2cb2bf22ee51747161b174946a WatchSource:0}: Error finding container 88572bdeb772a472aa0bee6906aadfb956914d2cb2bf22ee51747161b174946a: Status 404 returned error can't find the container with id 88572bdeb772a472aa0bee6906aadfb956914d2cb2bf22ee51747161b174946a Oct 30 18:05:54.855017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.854985 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:05:54.855090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.855054 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:05:54.855090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.855083 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:05:54.899869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.899837 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad529b6e-zk8jz\" (UID: \"c9444bfd-5672-413b-b80d-6b6bea511a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:54.900050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.899878 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad529b6e-zk8jz\" (UID: \"c9444bfd-5672-413b-b80d-6b6bea511a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:54.900050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:54.900024 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz2sg\" (UniqueName: \"kubernetes.io/projected/c9444bfd-5672-413b-b80d-6b6bea511a7f-kube-api-access-lz2sg\") pod \"maas-api-revoke-sub-ad529b6e-zk8jz\" (UID: \"c9444bfd-5672-413b-b80d-6b6bea511a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:55.001499 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.001452 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad529b6e-zk8jz\" (UID: \"c9444bfd-5672-413b-b80d-6b6bea511a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:55.001499 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.001498 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad529b6e-zk8jz\" (UID: \"c9444bfd-5672-413b-b80d-6b6bea511a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:55.001778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.001551 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz2sg\" (UniqueName: \"kubernetes.io/projected/c9444bfd-5672-413b-b80d-6b6bea511a7f-kube-api-access-lz2sg\") pod \"maas-api-revoke-sub-ad529b6e-zk8jz\" (UID: \"c9444bfd-5672-413b-b80d-6b6bea511a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:55.002640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.002604 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad529b6e-zk8jz\" (UID: \"c9444bfd-5672-413b-b80d-6b6bea511a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:55.004062 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.004038 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad529b6e-zk8jz\" (UID: \"c9444bfd-5672-413b-b80d-6b6bea511a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:55.028622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.028542 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz2sg\" (UniqueName: \"kubernetes.io/projected/c9444bfd-5672-413b-b80d-6b6bea511a7f-kube-api-access-lz2sg\") pod \"maas-api-revoke-sub-ad529b6e-zk8jz\" (UID: \"c9444bfd-5672-413b-b80d-6b6bea511a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:55.095200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.095165 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:55.225513 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.225455 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" event={"ID":"40b5a129-7409-43f5-8ffc-5c6b7738cdb4","Type":"ContainerStarted","Data":"c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379"} Oct 30 18:05:55.226009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.225568 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" event={"ID":"40b5a129-7409-43f5-8ffc-5c6b7738cdb4","Type":"ContainerStarted","Data":"88572bdeb772a472aa0bee6906aadfb956914d2cb2bf22ee51747161b174946a"} Oct 30 18:05:55.276417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.276364 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podStartSLOduration=1.276349862 podStartE2EDuration="1.276349862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:55.274162376 +0000 UTC m=+814.231166211" watchObservedRunningTime="2026-10-30 18:05:55.276349862 +0000 UTC m=+814.233353689" Oct 30 18:05:55.410257 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.410225 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz"] Oct 30 18:05:55.412394 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:05:55.412364 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9444bfd_5672_413b_b80d_6b6bea511a7f.slice/crio-09b2d52ae2e6f698a61cfd35c0427c316ec3002623eda9b672d5942a0ee81d54 WatchSource:0}: Error finding container 09b2d52ae2e6f698a61cfd35c0427c316ec3002623eda9b672d5942a0ee81d54: Status 404 returned error can't find the container with id 09b2d52ae2e6f698a61cfd35c0427c316ec3002623eda9b672d5942a0ee81d54 Oct 30 18:05:55.519822 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.519782 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:05:55.521784 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.521753 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:05:55.521913 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:55.521806 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:05:56.233986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:56.233916 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" event={"ID":"c9444bfd-5672-413b-b80d-6b6bea511a7f","Type":"ContainerStarted","Data":"c87a2dbfb14a4f91d0df1cf2a1ee25c8e2c91b17de9d03ad85fb6c5697f1a188"} Oct 30 18:05:56.234492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:56.234009 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" event={"ID":"c9444bfd-5672-413b-b80d-6b6bea511a7f","Type":"ContainerStarted","Data":"09b2d52ae2e6f698a61cfd35c0427c316ec3002623eda9b672d5942a0ee81d54"} Oct 30 18:05:56.291114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:56.291041 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" podStartSLOduration=2.2910182519999998 podStartE2EDuration="2.291018252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:56.289384922 +0000 UTC m=+815.246388748" watchObservedRunningTime="2026-10-30 18:05:56.291018252 +0000 UTC m=+815.248022089" Oct 30 18:05:56.520145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:56.520051 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:05:56.520322 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:56.520120 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:05:57.241389 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:57.241351 4883 generic.go:356] "Generic (PLEG): container finished" podID="c9444bfd-5672-413b-b80d-6b6bea511a7f" containerID="c87a2dbfb14a4f91d0df1cf2a1ee25c8e2c91b17de9d03ad85fb6c5697f1a188" exitCode=0 Oct 30 18:05:57.241904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:57.241435 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" event={"ID":"c9444bfd-5672-413b-b80d-6b6bea511a7f","Type":"ContainerDied","Data":"c87a2dbfb14a4f91d0df1cf2a1ee25c8e2c91b17de9d03ad85fb6c5697f1a188"} Oct 30 18:05:57.519909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:57.519822 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:05:57.519909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:57.519880 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:05:58.428904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:58.428870 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:58.520280 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:58.520191 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:05:58.520280 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:58.520257 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:05:58.544077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:58.544039 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-openshift-ca\") pod \"c9444bfd-5672-413b-b80d-6b6bea511a7f\" (UID: \"c9444bfd-5672-413b-b80d-6b6bea511a7f\") " Oct 30 18:05:58.544259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:58.544124 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-xks-ca\") pod \"c9444bfd-5672-413b-b80d-6b6bea511a7f\" (UID: \"c9444bfd-5672-413b-b80d-6b6bea511a7f\") " Oct 30 18:05:58.544350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:58.544286 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9444bfd-5672-413b-b80d-6b6bea511a7f-kube-api-access-lz2sg\" (UniqueName: \"kubernetes.io/projected/c9444bfd-5672-413b-b80d-6b6bea511a7f-kube-api-access-lz2sg\") pod \"c9444bfd-5672-413b-b80d-6b6bea511a7f\" (UID: \"c9444bfd-5672-413b-b80d-6b6bea511a7f\") " Oct 30 18:05:58.544408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:58.544388 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-openshift-ca" pod "c9444bfd-5672-413b-b80d-6b6bea511a7f" (UID: "c9444bfd-5672-413b-b80d-6b6bea511a7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:58.544588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:58.544572 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:05:58.546258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:58.546231 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-xks-ca" pod "c9444bfd-5672-413b-b80d-6b6bea511a7f" (UID: "c9444bfd-5672-413b-b80d-6b6bea511a7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:58.546380 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:58.546360 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9444bfd-5672-413b-b80d-6b6bea511a7f-kube-api-access-lz2sg" pod "c9444bfd-5672-413b-b80d-6b6bea511a7f" (UID: "c9444bfd-5672-413b-b80d-6b6bea511a7f"). InnerVolumeSpecName "kube-api-access-lz2sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:58.645563 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:58.645527 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz2sg\" (UniqueName: \"kubernetes.io/projected/c9444bfd-5672-413b-b80d-6b6bea511a7f-kube-api-access-lz2sg\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:05:58.645563 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:58.645559 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9444bfd-5672-413b-b80d-6b6bea511a7f-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:05:59.254396 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:59.254367 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" Oct 30 18:05:59.254396 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:59.254386 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz" event={"ID":"c9444bfd-5672-413b-b80d-6b6bea511a7f","Type":"ContainerDied","Data":"09b2d52ae2e6f698a61cfd35c0427c316ec3002623eda9b672d5942a0ee81d54"} Oct 30 18:05:59.254669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:59.254435 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09b2d52ae2e6f698a61cfd35c0427c316ec3002623eda9b672d5942a0ee81d54" Oct 30 18:05:59.519767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:59.519679 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:05:59.519767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:05:59.519744 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:06:00.519926 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:00.519888 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:06:00.520335 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:00.519976 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:06:01.519465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:01.519431 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:06:01.519658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:01.519497 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:06:02.524027 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:02.523994 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:06:02.524422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:02.524293 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:06:02.525006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:02.524986 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:06:05.108669 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:06:05.108573 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:06:10.607484 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.607441 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q"] Oct 30 18:06:10.607959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.607927 4883 cpu_manager.go:414] "containerMap: removing container" podUID="c9444bfd-5672-413b-b80d-6b6bea511a7f" containerName="revoke-keys" Oct 30 18:06:10.608008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.607962 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9444bfd-5672-413b-b80d-6b6bea511a7f" containerName="revoke-keys" Oct 30 18:06:10.608120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.608109 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9444bfd-5672-413b-b80d-6b6bea511a7f" containerName="revoke-keys" Oct 30 18:06:10.611565 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.611541 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:10.632490 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.632454 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:06:10.646185 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.646148 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:06:10.646416 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.646187 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:06:10.646416 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.646249 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:06:10.649131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.649097 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q"] Oct 30 18:06:10.713238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.713202 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mv2\" (UniqueName: \"kubernetes.io/projected/78c4ed74-4e02-41d8-84a2-bb662616dfe4-kube-api-access-x9mv2\") pod \"maas-api-revoke-sub-70f7174b-69c7q\" (UID: \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:10.713417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.713259 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70f7174b-69c7q\" (UID: \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:10.713417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.713347 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70f7174b-69c7q\" (UID: \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:10.814125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.814077 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70f7174b-69c7q\" (UID: \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:10.814125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.814124 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70f7174b-69c7q\" (UID: \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:10.814349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.814301 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mv2\" (UniqueName: \"kubernetes.io/projected/78c4ed74-4e02-41d8-84a2-bb662616dfe4-kube-api-access-x9mv2\") pod \"maas-api-revoke-sub-70f7174b-69c7q\" (UID: \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:10.815008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.814928 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70f7174b-69c7q\" (UID: \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:10.816613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.816593 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70f7174b-69c7q\" (UID: \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:10.839441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.839407 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mv2\" (UniqueName: \"kubernetes.io/projected/78c4ed74-4e02-41d8-84a2-bb662616dfe4-kube-api-access-x9mv2\") pod \"maas-api-revoke-sub-70f7174b-69c7q\" (UID: \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:10.926164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:10.926071 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:11.235237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:11.235203 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q"] Oct 30 18:06:11.236829 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:06:11.236802 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c4ed74_4e02_41d8_84a2_bb662616dfe4.slice/crio-0ec0e9b39bfab77f30d6361fa5bd7704f629d91269a8f9de51849177cb2de982 WatchSource:0}: Error finding container 0ec0e9b39bfab77f30d6361fa5bd7704f629d91269a8f9de51849177cb2de982: Status 404 returned error can't find the container with id 0ec0e9b39bfab77f30d6361fa5bd7704f629d91269a8f9de51849177cb2de982 Oct 30 18:06:11.338364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:11.338328 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" event={"ID":"78c4ed74-4e02-41d8-84a2-bb662616dfe4","Type":"ContainerStarted","Data":"0ec0e9b39bfab77f30d6361fa5bd7704f629d91269a8f9de51849177cb2de982"} Oct 30 18:06:12.345794 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:12.345764 4883 generic.go:356] "Generic (PLEG): container finished" podID="78c4ed74-4e02-41d8-84a2-bb662616dfe4" containerID="fd6ea32cdad4c430ff2cc6e44d50e51fd0d74d50ec1e1e6dd7970c789d34fa3e" exitCode=0 Oct 30 18:06:12.346205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:12.345852 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" event={"ID":"78c4ed74-4e02-41d8-84a2-bb662616dfe4","Type":"ContainerDied","Data":"fd6ea32cdad4c430ff2cc6e44d50e51fd0d74d50ec1e1e6dd7970c789d34fa3e"} Oct 30 18:06:13.493117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:13.493086 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:13.646324 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:13.646236 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c4ed74-4e02-41d8-84a2-bb662616dfe4-kube-api-access-x9mv2\" (UniqueName: \"kubernetes.io/projected/78c4ed74-4e02-41d8-84a2-bb662616dfe4-kube-api-access-x9mv2\") pod \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\" (UID: \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\") " Oct 30 18:06:13.646473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:13.646346 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-xks-ca\") pod \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\" (UID: \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\") " Oct 30 18:06:13.646473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:13.646374 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-openshift-ca\") pod \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\" (UID: \"78c4ed74-4e02-41d8-84a2-bb662616dfe4\") " Oct 30 18:06:13.646806 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:13.646775 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-openshift-ca" pod "78c4ed74-4e02-41d8-84a2-bb662616dfe4" (UID: "78c4ed74-4e02-41d8-84a2-bb662616dfe4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:13.648507 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:13.648480 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-xks-ca" pod "78c4ed74-4e02-41d8-84a2-bb662616dfe4" (UID: "78c4ed74-4e02-41d8-84a2-bb662616dfe4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:13.648507 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:13.648487 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c4ed74-4e02-41d8-84a2-bb662616dfe4-kube-api-access-x9mv2" pod "78c4ed74-4e02-41d8-84a2-bb662616dfe4" (UID: "78c4ed74-4e02-41d8-84a2-bb662616dfe4"). InnerVolumeSpecName "kube-api-access-x9mv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:13.748097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:13.748062 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:13.748097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:13.748090 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c4ed74-4e02-41d8-84a2-bb662616dfe4-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:13.748097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:13.748100 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9mv2\" (UniqueName: \"kubernetes.io/projected/78c4ed74-4e02-41d8-84a2-bb662616dfe4-kube-api-access-x9mv2\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:14.359435 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:14.359400 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" Oct 30 18:06:14.359435 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:14.359419 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q" event={"ID":"78c4ed74-4e02-41d8-84a2-bb662616dfe4","Type":"ContainerDied","Data":"0ec0e9b39bfab77f30d6361fa5bd7704f629d91269a8f9de51849177cb2de982"} Oct 30 18:06:14.359663 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:14.359507 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ec0e9b39bfab77f30d6361fa5bd7704f629d91269a8f9de51849177cb2de982" Oct 30 18:06:16.609045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.609006 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc"] Oct 30 18:06:16.609502 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.609471 4883 cpu_manager.go:414] "containerMap: removing container" podUID="78c4ed74-4e02-41d8-84a2-bb662616dfe4" containerName="revoke-keys" Oct 30 18:06:16.609502 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.609483 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c4ed74-4e02-41d8-84a2-bb662616dfe4" containerName="revoke-keys" Oct 30 18:06:16.609590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.609580 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c4ed74-4e02-41d8-84a2-bb662616dfe4" containerName="revoke-keys" Oct 30 18:06:16.613131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.613109 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:16.630053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.630025 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:06:16.643708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.643672 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:06:16.643708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.643680 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:06:16.643953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.643770 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:06:16.647864 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.647827 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc"] Oct 30 18:06:16.781001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.780961 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958c424a-xzxnc\" (UID: \"48e2f012-f684-480d-8f06-d91b6ae6020d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:16.781214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.781187 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck4cq\" (UniqueName: \"kubernetes.io/projected/48e2f012-f684-480d-8f06-d91b6ae6020d-kube-api-access-ck4cq\") pod \"maas-api-revoke-sub-958c424a-xzxnc\" (UID: \"48e2f012-f684-480d-8f06-d91b6ae6020d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:16.781330 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.781315 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958c424a-xzxnc\" (UID: \"48e2f012-f684-480d-8f06-d91b6ae6020d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:16.882177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.882081 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958c424a-xzxnc\" (UID: \"48e2f012-f684-480d-8f06-d91b6ae6020d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:16.882177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.882130 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958c424a-xzxnc\" (UID: \"48e2f012-f684-480d-8f06-d91b6ae6020d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:16.882410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.882218 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck4cq\" (UniqueName: \"kubernetes.io/projected/48e2f012-f684-480d-8f06-d91b6ae6020d-kube-api-access-ck4cq\") pod \"maas-api-revoke-sub-958c424a-xzxnc\" (UID: \"48e2f012-f684-480d-8f06-d91b6ae6020d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:16.883018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.882991 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958c424a-xzxnc\" (UID: \"48e2f012-f684-480d-8f06-d91b6ae6020d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:16.884591 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.884566 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958c424a-xzxnc\" (UID: \"48e2f012-f684-480d-8f06-d91b6ae6020d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:16.907302 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.907264 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck4cq\" (UniqueName: \"kubernetes.io/projected/48e2f012-f684-480d-8f06-d91b6ae6020d-kube-api-access-ck4cq\") pod \"maas-api-revoke-sub-958c424a-xzxnc\" (UID: \"48e2f012-f684-480d-8f06-d91b6ae6020d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:16.926797 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:16.926765 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:17.236282 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:17.236207 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc"] Oct 30 18:06:17.240042 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:06:17.240009 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e2f012_f684_480d_8f06_d91b6ae6020d.slice/crio-7555e03cb5502dad44a54858f0734a6a4c88d977e906e681844b01be01b29999 WatchSource:0}: Error finding container 7555e03cb5502dad44a54858f0734a6a4c88d977e906e681844b01be01b29999: Status 404 returned error can't find the container with id 7555e03cb5502dad44a54858f0734a6a4c88d977e906e681844b01be01b29999 Oct 30 18:06:17.383439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:17.383403 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" event={"ID":"48e2f012-f684-480d-8f06-d91b6ae6020d","Type":"ContainerStarted","Data":"7555e03cb5502dad44a54858f0734a6a4c88d977e906e681844b01be01b29999"} Oct 30 18:06:18.391047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:18.391010 4883 generic.go:356] "Generic (PLEG): container finished" podID="48e2f012-f684-480d-8f06-d91b6ae6020d" containerID="12270eb7f549d2274574a4ffa13e517a17f70d61591497643dd2f5046e4de758" exitCode=0 Oct 30 18:06:18.391459 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:18.391088 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" event={"ID":"48e2f012-f684-480d-8f06-d91b6ae6020d","Type":"ContainerDied","Data":"12270eb7f549d2274574a4ffa13e517a17f70d61591497643dd2f5046e4de758"} Oct 30 18:06:19.560604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:19.560575 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:19.718526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:19.718427 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e2f012-f684-480d-8f06-d91b6ae6020d-kube-api-access-ck4cq\" (UniqueName: \"kubernetes.io/projected/48e2f012-f684-480d-8f06-d91b6ae6020d-kube-api-access-ck4cq\") pod \"48e2f012-f684-480d-8f06-d91b6ae6020d\" (UID: \"48e2f012-f684-480d-8f06-d91b6ae6020d\") " Oct 30 18:06:19.718526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:19.718516 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-openshift-ca\") pod \"48e2f012-f684-480d-8f06-d91b6ae6020d\" (UID: \"48e2f012-f684-480d-8f06-d91b6ae6020d\") " Oct 30 18:06:19.718769 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:19.718569 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-xks-ca\") pod \"48e2f012-f684-480d-8f06-d91b6ae6020d\" (UID: \"48e2f012-f684-480d-8f06-d91b6ae6020d\") " Oct 30 18:06:19.718990 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:19.718965 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-openshift-ca" pod "48e2f012-f684-480d-8f06-d91b6ae6020d" (UID: "48e2f012-f684-480d-8f06-d91b6ae6020d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:19.720654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:19.720627 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-xks-ca" pod "48e2f012-f684-480d-8f06-d91b6ae6020d" (UID: "48e2f012-f684-480d-8f06-d91b6ae6020d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:19.720787 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:19.720766 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e2f012-f684-480d-8f06-d91b6ae6020d-kube-api-access-ck4cq" pod "48e2f012-f684-480d-8f06-d91b6ae6020d" (UID: "48e2f012-f684-480d-8f06-d91b6ae6020d"). InnerVolumeSpecName "kube-api-access-ck4cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:19.819823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:19.819781 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck4cq\" (UniqueName: \"kubernetes.io/projected/48e2f012-f684-480d-8f06-d91b6ae6020d-kube-api-access-ck4cq\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:19.819823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:19.819816 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:19.819823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:19.819826 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e2f012-f684-480d-8f06-d91b6ae6020d-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:20.402968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:20.402910 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" Oct 30 18:06:20.402968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:20.402926 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc" event={"ID":"48e2f012-f684-480d-8f06-d91b6ae6020d","Type":"ContainerDied","Data":"7555e03cb5502dad44a54858f0734a6a4c88d977e906e681844b01be01b29999"} Oct 30 18:06:20.403178 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:20.403014 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7555e03cb5502dad44a54858f0734a6a4c88d977e906e681844b01be01b29999" Oct 30 18:06:23.385765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.385726 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b"] Oct 30 18:06:23.386249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.386233 4883 cpu_manager.go:414] "containerMap: removing container" podUID="48e2f012-f684-480d-8f06-d91b6ae6020d" containerName="revoke-keys" Oct 30 18:06:23.386294 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.386250 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e2f012-f684-480d-8f06-d91b6ae6020d" containerName="revoke-keys" Oct 30 18:06:23.386329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.386320 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e2f012-f684-480d-8f06-d91b6ae6020d" containerName="revoke-keys" Oct 30 18:06:23.390061 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.390029 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:23.408771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.408718 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:06:23.408771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.408733 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:06:23.409023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.408793 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:06:23.422512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.422474 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:06:23.426122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.426089 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b"] Oct 30 18:06:23.459420 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.459377 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16135d4b-npj7b\" (UID: \"60e72e77-96d1-44bd-9085-b688edd838dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:23.459569 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.459548 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16135d4b-npj7b\" (UID: \"60e72e77-96d1-44bd-9085-b688edd838dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:23.459633 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.459580 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m87pm\" (UniqueName: \"kubernetes.io/projected/60e72e77-96d1-44bd-9085-b688edd838dc-kube-api-access-m87pm\") pod \"maas-api-revoke-sub-16135d4b-npj7b\" (UID: \"60e72e77-96d1-44bd-9085-b688edd838dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:23.560974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.560900 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16135d4b-npj7b\" (UID: \"60e72e77-96d1-44bd-9085-b688edd838dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:23.561176 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.561034 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16135d4b-npj7b\" (UID: \"60e72e77-96d1-44bd-9085-b688edd838dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:23.561176 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.561058 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m87pm\" (UniqueName: \"kubernetes.io/projected/60e72e77-96d1-44bd-9085-b688edd838dc-kube-api-access-m87pm\") pod \"maas-api-revoke-sub-16135d4b-npj7b\" (UID: \"60e72e77-96d1-44bd-9085-b688edd838dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:23.561742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.561713 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16135d4b-npj7b\" (UID: \"60e72e77-96d1-44bd-9085-b688edd838dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:23.563517 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.563485 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16135d4b-npj7b\" (UID: \"60e72e77-96d1-44bd-9085-b688edd838dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:23.586131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.586098 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m87pm\" (UniqueName: \"kubernetes.io/projected/60e72e77-96d1-44bd-9085-b688edd838dc-kube-api-access-m87pm\") pod \"maas-api-revoke-sub-16135d4b-npj7b\" (UID: \"60e72e77-96d1-44bd-9085-b688edd838dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:23.703815 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:23.703728 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:24.019085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:24.019055 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b"] Oct 30 18:06:24.020382 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:06:24.020354 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e72e77_96d1_44bd_9085_b688edd838dc.slice/crio-ee4aeeb8a6a64889c2f7fedf86ac64c82a88a9eb6ea3aad2e2e99d6089576b86 WatchSource:0}: Error finding container ee4aeeb8a6a64889c2f7fedf86ac64c82a88a9eb6ea3aad2e2e99d6089576b86: Status 404 returned error can't find the container with id ee4aeeb8a6a64889c2f7fedf86ac64c82a88a9eb6ea3aad2e2e99d6089576b86 Oct 30 18:06:24.436990 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:24.436910 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" event={"ID":"60e72e77-96d1-44bd-9085-b688edd838dc","Type":"ContainerStarted","Data":"eaf1c99a76cd5302c7d8466821af1ddc0b3589b3b5044d33eca10a3c96f3a63d"} Oct 30 18:06:24.437438 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:24.437006 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" event={"ID":"60e72e77-96d1-44bd-9085-b688edd838dc","Type":"ContainerStarted","Data":"ee4aeeb8a6a64889c2f7fedf86ac64c82a88a9eb6ea3aad2e2e99d6089576b86"} Oct 30 18:06:24.484416 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:24.484364 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" podStartSLOduration=1.4843456289999999 podStartE2EDuration="1.484345629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:24.482487573 +0000 UTC m=+843.439491408" watchObservedRunningTime="2026-10-30 18:06:24.484345629 +0000 UTC m=+843.441349465" Oct 30 18:06:25.445319 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:25.445279 4883 generic.go:356] "Generic (PLEG): container finished" podID="60e72e77-96d1-44bd-9085-b688edd838dc" containerID="eaf1c99a76cd5302c7d8466821af1ddc0b3589b3b5044d33eca10a3c96f3a63d" exitCode=0 Oct 30 18:06:25.445904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:25.445365 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" event={"ID":"60e72e77-96d1-44bd-9085-b688edd838dc","Type":"ContainerDied","Data":"eaf1c99a76cd5302c7d8466821af1ddc0b3589b3b5044d33eca10a3c96f3a63d"} Oct 30 18:06:26.631432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:26.631395 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:26.696978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:26.696908 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-openshift-ca\") pod \"60e72e77-96d1-44bd-9085-b688edd838dc\" (UID: \"60e72e77-96d1-44bd-9085-b688edd838dc\") " Oct 30 18:06:26.696978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:26.696964 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e72e77-96d1-44bd-9085-b688edd838dc-kube-api-access-m87pm\" (UniqueName: \"kubernetes.io/projected/60e72e77-96d1-44bd-9085-b688edd838dc-kube-api-access-m87pm\") pod \"60e72e77-96d1-44bd-9085-b688edd838dc\" (UID: \"60e72e77-96d1-44bd-9085-b688edd838dc\") " Oct 30 18:06:26.697229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:26.697036 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-xks-ca\") pod \"60e72e77-96d1-44bd-9085-b688edd838dc\" (UID: \"60e72e77-96d1-44bd-9085-b688edd838dc\") " Oct 30 18:06:26.697368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:26.697335 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-openshift-ca" pod "60e72e77-96d1-44bd-9085-b688edd838dc" (UID: "60e72e77-96d1-44bd-9085-b688edd838dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:26.699403 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:26.699371 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-xks-ca" pod "60e72e77-96d1-44bd-9085-b688edd838dc" (UID: "60e72e77-96d1-44bd-9085-b688edd838dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:26.699530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:26.699469 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e72e77-96d1-44bd-9085-b688edd838dc-kube-api-access-m87pm" pod "60e72e77-96d1-44bd-9085-b688edd838dc" (UID: "60e72e77-96d1-44bd-9085-b688edd838dc"). InnerVolumeSpecName "kube-api-access-m87pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:26.798100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:26.798068 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:26.798100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:26.798096 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m87pm\" (UniqueName: \"kubernetes.io/projected/60e72e77-96d1-44bd-9085-b688edd838dc-kube-api-access-m87pm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:26.798100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:26.798107 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e72e77-96d1-44bd-9085-b688edd838dc-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:27.458742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:27.458653 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" Oct 30 18:06:27.458742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:27.458691 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b" event={"ID":"60e72e77-96d1-44bd-9085-b688edd838dc","Type":"ContainerDied","Data":"ee4aeeb8a6a64889c2f7fedf86ac64c82a88a9eb6ea3aad2e2e99d6089576b86"} Oct 30 18:06:27.458742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:27.458739 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee4aeeb8a6a64889c2f7fedf86ac64c82a88a9eb6ea3aad2e2e99d6089576b86" Oct 30 18:06:28.689613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.689576 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8"] Oct 30 18:06:28.690306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.690285 4883 cpu_manager.go:414] "containerMap: removing container" podUID="60e72e77-96d1-44bd-9085-b688edd838dc" containerName="revoke-keys" Oct 30 18:06:28.690412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.690306 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e72e77-96d1-44bd-9085-b688edd838dc" containerName="revoke-keys" Oct 30 18:06:28.690468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.690420 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e72e77-96d1-44bd-9085-b688edd838dc" containerName="revoke-keys" Oct 30 18:06:28.693866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.693825 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.721016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.716222 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.721016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.716279 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.721016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.716326 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60201992-7051-4dbc-8798-ef7ae9152518-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.721016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.716354 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ts9p\" (UniqueName: \"kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-kube-api-access-7ts9p\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.721016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.716403 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.721016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.716427 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.721016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.716488 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60201992-7051-4dbc-8798-ef7ae9152518-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.721016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.716605 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.721016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.716647 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.728718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.728686 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8"] Oct 30 18:06:28.822626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.822585 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60201992-7051-4dbc-8798-ef7ae9152518-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.822987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.822965 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.823134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.823121 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.823318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.823305 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.823428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.823414 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.823541 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.823529 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60201992-7051-4dbc-8798-ef7ae9152518-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.823636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.823624 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ts9p\" (UniqueName: \"kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-kube-api-access-7ts9p\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.823750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.823736 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.823845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.823832 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.824498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.824472 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-data\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.824696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.824659 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.824777 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.824733 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-certs\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.825430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.825079 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-credential-socket\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.825430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.825152 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60201992-7051-4dbc-8798-ef7ae9152518-istiod-ca-cert\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.827492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.827449 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60201992-7051-4dbc-8798-ef7ae9152518-istio-podinfo\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.829364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.829337 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-envoy\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.849219 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.849182 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-istio-token\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:28.849375 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:28.849269 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ts9p\" (UniqueName: \"kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-kube-api-access-7ts9p\") pod \"e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:29.012394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:29.012296 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:29.338804 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:29.338771 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8"] Oct 30 18:06:29.340647 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:06:29.340621 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60201992_7051_4dbc_8798_ef7ae9152518.slice/crio-0c396a28c223dee92f3657cdbe02a1e31ef52cd7b1269702e4ca706f4ec27cc4 WatchSource:0}: Error finding container 0c396a28c223dee92f3657cdbe02a1e31ef52cd7b1269702e4ca706f4ec27cc4: Status 404 returned error can't find the container with id 0c396a28c223dee92f3657cdbe02a1e31ef52cd7b1269702e4ca706f4ec27cc4 Oct 30 18:06:29.342985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:29.342928 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:06:29.343082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:29.343033 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:06:29.343149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:29.343085 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:06:29.472615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:29.472579 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" event={"ID":"60201992-7051-4dbc-8798-ef7ae9152518","Type":"ContainerStarted","Data":"8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218"} Oct 30 18:06:29.472797 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:29.472637 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" event={"ID":"60201992-7051-4dbc-8798-ef7ae9152518","Type":"ContainerStarted","Data":"0c396a28c223dee92f3657cdbe02a1e31ef52cd7b1269702e4ca706f4ec27cc4"} Oct 30 18:06:29.522545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:29.522483 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podStartSLOduration=1.522461758 podStartE2EDuration="1.522461758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:29.51944901 +0000 UTC m=+848.476452846" watchObservedRunningTime="2026-10-30 18:06:29.522461758 +0000 UTC m=+848.479465598" Oct 30 18:06:30.013231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:30.013188 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:30.014963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:30.014901 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:06:30.015102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:30.014964 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:06:31.013602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.013560 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:06:31.014075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.013634 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:06:31.359088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.359048 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr"] Oct 30 18:06:31.363364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.363339 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:31.396976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.396548 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr"] Oct 30 18:06:31.406453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.406424 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:06:31.406600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.406521 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:06:31.406600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.406570 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:06:31.406676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.406663 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:06:31.412092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.412032 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs"] Oct 30 18:06:31.416396 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.416367 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.444542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.444499 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs"] Oct 30 18:06:31.453830 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.453792 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0789a37e-7327-47e9-9bc1-247533139ae4-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.453830 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.453834 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.454133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.453866 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.454133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.454091 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trtx\" (UniqueName: \"kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-kube-api-access-4trtx\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.454214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.454155 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98bf06ca-xlnjr\" (UID: \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:31.454214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.454190 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0789a37e-7327-47e9-9bc1-247533139ae4-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.454297 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.454280 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.454334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.454312 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.454415 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.454396 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.454448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.454437 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bkh6\" (UniqueName: \"kubernetes.io/projected/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-kube-api-access-7bkh6\") pod \"maas-api-revoke-sub-98bf06ca-xlnjr\" (UID: \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:31.454582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.454558 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.454626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.454606 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98bf06ca-xlnjr\" (UID: \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:31.556252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556202 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.556460 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556372 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trtx\" (UniqueName: \"kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-kube-api-access-4trtx\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.556460 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556455 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98bf06ca-xlnjr\" (UID: \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:31.556581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556479 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0789a37e-7327-47e9-9bc1-247533139ae4-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.556581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556533 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.556683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556600 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.556683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556627 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.556683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556633 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.556683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556655 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bkh6\" (UniqueName: \"kubernetes.io/projected/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-kube-api-access-7bkh6\") pod \"maas-api-revoke-sub-98bf06ca-xlnjr\" (UID: \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:31.556683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556681 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.557024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556701 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98bf06ca-xlnjr\" (UID: \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:31.557024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556797 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0789a37e-7327-47e9-9bc1-247533139ae4-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.557024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.556817 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.557192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.557106 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.557286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.557251 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0789a37e-7327-47e9-9bc1-247533139ae4-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.557343 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.557311 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.557401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.557384 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.557526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.557457 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98bf06ca-xlnjr\" (UID: \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:31.559235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.559209 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.559724 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.559707 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0789a37e-7327-47e9-9bc1-247533139ae4-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.559773 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.559699 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98bf06ca-xlnjr\" (UID: \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:31.594593 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.594550 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.594772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.594681 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bkh6\" (UniqueName: \"kubernetes.io/projected/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-kube-api-access-7bkh6\") pod \"maas-api-revoke-sub-98bf06ca-xlnjr\" (UID: \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:31.596316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.596290 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trtx\" (UniqueName: \"kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-kube-api-access-4trtx\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:31.709867 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.709781 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:31.734044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:31.734009 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:32.013752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.013667 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:06:32.013752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.013725 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:06:32.044805 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.044777 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr"] Oct 30 18:06:32.046676 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:06:32.046646 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb444d7_c764_4f6c_bdbf_aeca289c89c0.slice/crio-3989598415eb0cadba14013eb2de775293dc8e65c4bcea3b066e0034474d5800 WatchSource:0}: Error finding container 3989598415eb0cadba14013eb2de775293dc8e65c4bcea3b066e0034474d5800: Status 404 returned error can't find the container with id 3989598415eb0cadba14013eb2de775293dc8e65c4bcea3b066e0034474d5800 Oct 30 18:06:32.093749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.093666 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs"] Oct 30 18:06:32.101723 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:06:32.101693 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0789a37e_7327_47e9_9bc1_247533139ae4.slice/crio-d1dacc40e1f2ad9aa8c7c924637fe8133c7c780149b75bf850e734db16886b85 WatchSource:0}: Error finding container d1dacc40e1f2ad9aa8c7c924637fe8133c7c780149b75bf850e734db16886b85: Status 404 returned error can't find the container with id d1dacc40e1f2ad9aa8c7c924637fe8133c7c780149b75bf850e734db16886b85 Oct 30 18:06:32.104664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.104627 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:06:32.104795 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.104781 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:06:32.104896 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.104873 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:06:32.516156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.515526 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" event={"ID":"0789a37e-7327-47e9-9bc1-247533139ae4","Type":"ContainerStarted","Data":"5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46"} Oct 30 18:06:32.516156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.515612 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" event={"ID":"0789a37e-7327-47e9-9bc1-247533139ae4","Type":"ContainerStarted","Data":"d1dacc40e1f2ad9aa8c7c924637fe8133c7c780149b75bf850e734db16886b85"} Oct 30 18:06:32.518730 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.518328 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" event={"ID":"ddb444d7-c764-4f6c-bdbf-aeca289c89c0","Type":"ContainerStarted","Data":"2449edce47acdaf79a7a220c4732a53faee639595d481ab9cb5014e03ed28842"} Oct 30 18:06:32.518730 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.518381 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" event={"ID":"ddb444d7-c764-4f6c-bdbf-aeca289c89c0","Type":"ContainerStarted","Data":"3989598415eb0cadba14013eb2de775293dc8e65c4bcea3b066e0034474d5800"} Oct 30 18:06:32.563925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.563856 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" podStartSLOduration=1.563840194 podStartE2EDuration="1.563840194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:32.561292188 +0000 UTC m=+851.518296025" watchObservedRunningTime="2026-10-30 18:06:32.563840194 +0000 UTC m=+851.520844029" Oct 30 18:06:32.734759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:32.734664 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:33.014681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:33.014537 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:06:33.015293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:33.015245 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:06:33.527005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:33.526967 4883 generic.go:356] "Generic (PLEG): container finished" podID="ddb444d7-c764-4f6c-bdbf-aeca289c89c0" containerID="2449edce47acdaf79a7a220c4732a53faee639595d481ab9cb5014e03ed28842" exitCode=0 Oct 30 18:06:33.527186 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:33.527054 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" event={"ID":"ddb444d7-c764-4f6c-bdbf-aeca289c89c0","Type":"ContainerDied","Data":"2449edce47acdaf79a7a220c4732a53faee639595d481ab9cb5014e03ed28842"} Oct 30 18:06:33.736719 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:33.736626 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:06:33.736719 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:33.736688 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 18:06:34.014025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:34.013383 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:06:34.014025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:34.013455 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:06:34.736085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:34.736044 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:06:34.736567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:34.736113 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 18:06:34.859986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:34.859957 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:35.014597 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.014427 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:06:35.014597 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.014485 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:06:35.044352 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.044315 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-xks-ca\") pod \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\" (UID: \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\") " Oct 30 18:06:35.044552 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.044380 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-openshift-ca\") pod \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\" (UID: \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\") " Oct 30 18:06:35.044552 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.044502 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-kube-api-access-7bkh6\" (UniqueName: \"kubernetes.io/projected/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-kube-api-access-7bkh6\") pod \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\" (UID: \"ddb444d7-c764-4f6c-bdbf-aeca289c89c0\") " Oct 30 18:06:35.044740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.044713 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-openshift-ca" pod "ddb444d7-c764-4f6c-bdbf-aeca289c89c0" (UID: "ddb444d7-c764-4f6c-bdbf-aeca289c89c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:35.044912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.044898 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:35.046603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.046576 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-kube-api-access-7bkh6" pod "ddb444d7-c764-4f6c-bdbf-aeca289c89c0" (UID: "ddb444d7-c764-4f6c-bdbf-aeca289c89c0"). InnerVolumeSpecName "kube-api-access-7bkh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:35.051752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.051719 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-xks-ca" pod "ddb444d7-c764-4f6c-bdbf-aeca289c89c0" (UID: "ddb444d7-c764-4f6c-bdbf-aeca289c89c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:35.124125 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:06:35.124060 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:06:35.145997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.145903 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:35.145997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.145968 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bkh6\" (UniqueName: \"kubernetes.io/projected/ddb444d7-c764-4f6c-bdbf-aeca289c89c0-kube-api-access-7bkh6\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:35.540868 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.540780 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" Oct 30 18:06:35.540868 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.540801 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr" event={"ID":"ddb444d7-c764-4f6c-bdbf-aeca289c89c0","Type":"ContainerDied","Data":"3989598415eb0cadba14013eb2de775293dc8e65c4bcea3b066e0034474d5800"} Oct 30 18:06:35.540868 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.540851 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3989598415eb0cadba14013eb2de775293dc8e65c4bcea3b066e0034474d5800" Oct 30 18:06:35.734739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.734694 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:06:35.734917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:35.734760 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 18:06:36.013503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:36.013464 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:06:36.013984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:36.013540 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:06:36.734834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:36.734793 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:06:36.735079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:36.734864 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 18:06:37.018185 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:37.018080 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:37.018601 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:37.018475 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:37.019345 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:37.019324 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:06:37.737092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:37.737049 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:06:37.737283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:37.737107 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 18:06:38.741960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:38.741453 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:06:38.741960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:38.741545 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 18:06:40.105956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:40.105467 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:40.105956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:40.105911 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:40.112585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:40.112555 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:06:41.572115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.572078 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6"] Oct 30 18:06:41.572587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.572573 4883 cpu_manager.go:414] "containerMap: removing container" podUID="ddb444d7-c764-4f6c-bdbf-aeca289c89c0" containerName="revoke-keys" Oct 30 18:06:41.572629 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.572588 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb444d7-c764-4f6c-bdbf-aeca289c89c0" containerName="revoke-keys" Oct 30 18:06:41.572672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.572662 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb444d7-c764-4f6c-bdbf-aeca289c89c0" containerName="revoke-keys" Oct 30 18:06:41.575517 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.575492 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:41.594156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.594099 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:06:41.607626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.607588 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:06:41.607626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.607588 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:06:41.607888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.607645 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:06:41.611540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.611505 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6"] Oct 30 18:06:41.633147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.633107 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59b11feb-gvwn6\" (UID: \"8ea47f43-d2d2-47c6-a183-99297339b3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:41.633362 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.633299 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69kc7\" (UniqueName: \"kubernetes.io/projected/8ea47f43-d2d2-47c6-a183-99297339b3f4-kube-api-access-69kc7\") pod \"maas-api-revoke-sub-59b11feb-gvwn6\" (UID: \"8ea47f43-d2d2-47c6-a183-99297339b3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:41.633457 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.633435 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59b11feb-gvwn6\" (UID: \"8ea47f43-d2d2-47c6-a183-99297339b3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:41.735184 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.735140 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59b11feb-gvwn6\" (UID: \"8ea47f43-d2d2-47c6-a183-99297339b3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:41.735360 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.735238 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69kc7\" (UniqueName: \"kubernetes.io/projected/8ea47f43-d2d2-47c6-a183-99297339b3f4-kube-api-access-69kc7\") pod \"maas-api-revoke-sub-59b11feb-gvwn6\" (UID: \"8ea47f43-d2d2-47c6-a183-99297339b3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:41.735360 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.735310 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59b11feb-gvwn6\" (UID: \"8ea47f43-d2d2-47c6-a183-99297339b3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:41.735992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.735971 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59b11feb-gvwn6\" (UID: \"8ea47f43-d2d2-47c6-a183-99297339b3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:41.737831 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.737803 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59b11feb-gvwn6\" (UID: \"8ea47f43-d2d2-47c6-a183-99297339b3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:41.761002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.760960 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69kc7\" (UniqueName: \"kubernetes.io/projected/8ea47f43-d2d2-47c6-a183-99297339b3f4-kube-api-access-69kc7\") pod \"maas-api-revoke-sub-59b11feb-gvwn6\" (UID: \"8ea47f43-d2d2-47c6-a183-99297339b3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:41.889549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:41.889508 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:42.197643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:42.197614 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6"] Oct 30 18:06:42.200074 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:06:42.200043 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea47f43_d2d2_47c6_a183_99297339b3f4.slice/crio-2408d9c2c60e34956b8a9200dc9541567089c23fa554c2808a30ef48cefce7ea WatchSource:0}: Error finding container 2408d9c2c60e34956b8a9200dc9541567089c23fa554c2808a30ef48cefce7ea: Status 404 returned error can't find the container with id 2408d9c2c60e34956b8a9200dc9541567089c23fa554c2808a30ef48cefce7ea Oct 30 18:06:42.621385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:42.621350 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" event={"ID":"8ea47f43-d2d2-47c6-a183-99297339b3f4","Type":"ContainerStarted","Data":"0a82d1e39bc741a9b5cd62a18f808f7bc0cf6248e6af6e61c8c57b94e64dc104"} Oct 30 18:06:42.621826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:42.621405 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" event={"ID":"8ea47f43-d2d2-47c6-a183-99297339b3f4","Type":"ContainerStarted","Data":"2408d9c2c60e34956b8a9200dc9541567089c23fa554c2808a30ef48cefce7ea"} Oct 30 18:06:42.672023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:42.671958 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" podStartSLOduration=1.67192718 podStartE2EDuration="1.67192718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:42.668975486 +0000 UTC m=+861.625979322" watchObservedRunningTime="2026-10-30 18:06:42.67192718 +0000 UTC m=+861.628931097" Oct 30 18:06:43.628843 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:43.628805 4883 generic.go:356] "Generic (PLEG): container finished" podID="8ea47f43-d2d2-47c6-a183-99297339b3f4" containerID="0a82d1e39bc741a9b5cd62a18f808f7bc0cf6248e6af6e61c8c57b94e64dc104" exitCode=0 Oct 30 18:06:43.629280 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:43.628897 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" event={"ID":"8ea47f43-d2d2-47c6-a183-99297339b3f4","Type":"ContainerDied","Data":"0a82d1e39bc741a9b5cd62a18f808f7bc0cf6248e6af6e61c8c57b94e64dc104"} Oct 30 18:06:44.804716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:44.804685 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:44.873386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:44.873344 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-xks-ca\") pod \"8ea47f43-d2d2-47c6-a183-99297339b3f4\" (UID: \"8ea47f43-d2d2-47c6-a183-99297339b3f4\") " Oct 30 18:06:44.873623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:44.873509 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-openshift-ca\") pod \"8ea47f43-d2d2-47c6-a183-99297339b3f4\" (UID: \"8ea47f43-d2d2-47c6-a183-99297339b3f4\") " Oct 30 18:06:44.873700 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:44.873641 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea47f43-d2d2-47c6-a183-99297339b3f4-kube-api-access-69kc7\" (UniqueName: \"kubernetes.io/projected/8ea47f43-d2d2-47c6-a183-99297339b3f4-kube-api-access-69kc7\") pod \"8ea47f43-d2d2-47c6-a183-99297339b3f4\" (UID: \"8ea47f43-d2d2-47c6-a183-99297339b3f4\") " Oct 30 18:06:44.874081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:44.874043 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-openshift-ca" pod "8ea47f43-d2d2-47c6-a183-99297339b3f4" (UID: "8ea47f43-d2d2-47c6-a183-99297339b3f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:44.875620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:44.875586 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-xks-ca" pod "8ea47f43-d2d2-47c6-a183-99297339b3f4" (UID: "8ea47f43-d2d2-47c6-a183-99297339b3f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:44.876473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:44.876446 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea47f43-d2d2-47c6-a183-99297339b3f4-kube-api-access-69kc7" pod "8ea47f43-d2d2-47c6-a183-99297339b3f4" (UID: "8ea47f43-d2d2-47c6-a183-99297339b3f4"). InnerVolumeSpecName "kube-api-access-69kc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:44.975171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:44.975079 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69kc7\" (UniqueName: \"kubernetes.io/projected/8ea47f43-d2d2-47c6-a183-99297339b3f4-kube-api-access-69kc7\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:44.975171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:44.975122 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:44.975171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:44.975139 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea47f43-d2d2-47c6-a183-99297339b3f4-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:45.643538 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:45.643500 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" event={"ID":"8ea47f43-d2d2-47c6-a183-99297339b3f4","Type":"ContainerDied","Data":"2408d9c2c60e34956b8a9200dc9541567089c23fa554c2808a30ef48cefce7ea"} Oct 30 18:06:45.643714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:45.643561 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2408d9c2c60e34956b8a9200dc9541567089c23fa554c2808a30ef48cefce7ea" Oct 30 18:06:45.643714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:45.643629 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6" Oct 30 18:06:51.924915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.924871 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9"] Oct 30 18:06:51.925450 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.925433 4883 cpu_manager.go:414] "containerMap: removing container" podUID="8ea47f43-d2d2-47c6-a183-99297339b3f4" containerName="revoke-keys" Oct 30 18:06:51.925489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.925452 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea47f43-d2d2-47c6-a183-99297339b3f4" containerName="revoke-keys" Oct 30 18:06:51.925567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.925558 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea47f43-d2d2-47c6-a183-99297339b3f4" containerName="revoke-keys" Oct 30 18:06:51.930716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.930521 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:51.946889 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.946838 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:06:51.960843 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.960789 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:06:51.960843 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.960804 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:06:51.961274 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.961246 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:06:51.964310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.964281 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9"] Oct 30 18:06:51.972602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.972568 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9f3fa4-pm9q9\" (UID: \"0f3941d7-23ca-4d49-b979-15fb991148c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:51.972852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.972825 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9f3fa4-pm9q9\" (UID: \"0f3941d7-23ca-4d49-b979-15fb991148c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:51.973007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:51.972920 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjj5g\" (UniqueName: \"kubernetes.io/projected/0f3941d7-23ca-4d49-b979-15fb991148c7-kube-api-access-jjj5g\") pod \"maas-api-revoke-sub-de9f3fa4-pm9q9\" (UID: \"0f3941d7-23ca-4d49-b979-15fb991148c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:52.073870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:52.073822 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9f3fa4-pm9q9\" (UID: \"0f3941d7-23ca-4d49-b979-15fb991148c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:52.073870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:52.073855 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9f3fa4-pm9q9\" (UID: \"0f3941d7-23ca-4d49-b979-15fb991148c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:52.074118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:52.073922 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjj5g\" (UniqueName: \"kubernetes.io/projected/0f3941d7-23ca-4d49-b979-15fb991148c7-kube-api-access-jjj5g\") pod \"maas-api-revoke-sub-de9f3fa4-pm9q9\" (UID: \"0f3941d7-23ca-4d49-b979-15fb991148c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:52.074659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:52.074633 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9f3fa4-pm9q9\" (UID: \"0f3941d7-23ca-4d49-b979-15fb991148c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:52.076599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:52.076573 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9f3fa4-pm9q9\" (UID: \"0f3941d7-23ca-4d49-b979-15fb991148c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:52.098978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:52.098929 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjj5g\" (UniqueName: \"kubernetes.io/projected/0f3941d7-23ca-4d49-b979-15fb991148c7-kube-api-access-jjj5g\") pod \"maas-api-revoke-sub-de9f3fa4-pm9q9\" (UID: \"0f3941d7-23ca-4d49-b979-15fb991148c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:52.243904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:52.243802 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:52.549819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:52.549789 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9"] Oct 30 18:06:52.552145 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:06:52.552117 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3941d7_23ca_4d49_b979_15fb991148c7.slice/crio-f76dd258234455fabb0075a052f59438374576c553ee3127ad2c39c16f07d353 WatchSource:0}: Error finding container f76dd258234455fabb0075a052f59438374576c553ee3127ad2c39c16f07d353: Status 404 returned error can't find the container with id f76dd258234455fabb0075a052f59438374576c553ee3127ad2c39c16f07d353 Oct 30 18:06:52.693504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:52.693469 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" event={"ID":"0f3941d7-23ca-4d49-b979-15fb991148c7","Type":"ContainerStarted","Data":"f76dd258234455fabb0075a052f59438374576c553ee3127ad2c39c16f07d353"} Oct 30 18:06:53.701865 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:53.701827 4883 generic.go:356] "Generic (PLEG): container finished" podID="0f3941d7-23ca-4d49-b979-15fb991148c7" containerID="045585b5100006868d9889226dd3823a8a0ead2de6fae25d1808d89de6a882c6" exitCode=0 Oct 30 18:06:53.702317 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:53.701916 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" event={"ID":"0f3941d7-23ca-4d49-b979-15fb991148c7","Type":"ContainerDied","Data":"045585b5100006868d9889226dd3823a8a0ead2de6fae25d1808d89de6a882c6"} Oct 30 18:06:54.881255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:54.881223 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:55.015598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:55.015499 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-openshift-ca\") pod \"0f3941d7-23ca-4d49-b979-15fb991148c7\" (UID: \"0f3941d7-23ca-4d49-b979-15fb991148c7\") " Oct 30 18:06:55.015789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:55.015672 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3941d7-23ca-4d49-b979-15fb991148c7-kube-api-access-jjj5g\" (UniqueName: \"kubernetes.io/projected/0f3941d7-23ca-4d49-b979-15fb991148c7-kube-api-access-jjj5g\") pod \"0f3941d7-23ca-4d49-b979-15fb991148c7\" (UID: \"0f3941d7-23ca-4d49-b979-15fb991148c7\") " Oct 30 18:06:55.015789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:55.015702 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-xks-ca\") pod \"0f3941d7-23ca-4d49-b979-15fb991148c7\" (UID: \"0f3941d7-23ca-4d49-b979-15fb991148c7\") " Oct 30 18:06:55.015910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:55.015874 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-openshift-ca" pod "0f3941d7-23ca-4d49-b979-15fb991148c7" (UID: "0f3941d7-23ca-4d49-b979-15fb991148c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:55.016055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:55.016033 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:55.018049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:55.018014 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3941d7-23ca-4d49-b979-15fb991148c7-kube-api-access-jjj5g" pod "0f3941d7-23ca-4d49-b979-15fb991148c7" (UID: "0f3941d7-23ca-4d49-b979-15fb991148c7"). InnerVolumeSpecName "kube-api-access-jjj5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:55.018163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:55.018050 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-xks-ca" pod "0f3941d7-23ca-4d49-b979-15fb991148c7" (UID: "0f3941d7-23ca-4d49-b979-15fb991148c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:55.116950 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:55.116907 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjj5g\" (UniqueName: \"kubernetes.io/projected/0f3941d7-23ca-4d49-b979-15fb991148c7-kube-api-access-jjj5g\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:55.117129 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:55.116981 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3941d7-23ca-4d49-b979-15fb991148c7-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:06:55.716266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:55.716237 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" Oct 30 18:06:55.716441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:55.716271 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9" event={"ID":"0f3941d7-23ca-4d49-b979-15fb991148c7","Type":"ContainerDied","Data":"f76dd258234455fabb0075a052f59438374576c553ee3127ad2c39c16f07d353"} Oct 30 18:06:55.716441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:06:55.716319 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f76dd258234455fabb0075a052f59438374576c553ee3127ad2c39c16f07d353" Oct 30 18:07:04.420088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.420051 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r"] Oct 30 18:07:04.420851 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.420819 4883 cpu_manager.go:414] "containerMap: removing container" podUID="0f3941d7-23ca-4d49-b979-15fb991148c7" containerName="revoke-keys" Oct 30 18:07:04.420851 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.420846 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3941d7-23ca-4d49-b979-15fb991148c7" containerName="revoke-keys" Oct 30 18:07:04.421549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.420997 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3941d7-23ca-4d49-b979-15fb991148c7" containerName="revoke-keys" Oct 30 18:07:04.425716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.425691 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.453969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.453916 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r"] Oct 30 18:07:04.528965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.528892 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.529138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.529030 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whnw9\" (UniqueName: \"kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-kube-api-access-whnw9\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.529138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.529060 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.529138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.529124 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.529250 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.529141 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357860ec-9d4e-4d5e-be19-eec58835b3dc-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.529250 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.529164 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.529312 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.529259 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.529402 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.529377 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.529487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.529471 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.630925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.630884 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whnw9\" (UniqueName: \"kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-kube-api-access-whnw9\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.630925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.630926 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.631277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.630995 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.631277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.631018 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357860ec-9d4e-4d5e-be19-eec58835b3dc-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.631277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.631169 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.631277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.631268 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.631537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.631372 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.631537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.631453 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.631537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.631529 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.631709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.631619 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.631910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.631880 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357860ec-9d4e-4d5e-be19-eec58835b3dc-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.631910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.631898 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.632140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.631904 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.632140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.632038 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.633855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.633830 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.634101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.634083 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.655833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.655793 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.655993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.655911 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whnw9\" (UniqueName: \"kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-kube-api-access-whnw9\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:04.740807 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:04.740720 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:05.073126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:05.073087 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r"] Oct 30 18:07:05.074854 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:07:05.074810 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357860ec_9d4e_4d5e_be19_eec58835b3dc.slice/crio-3adfc47fa1103f29ce818a0165f08dac3384594c18f1e76e8619ec81b816b157 WatchSource:0}: Error finding container 3adfc47fa1103f29ce818a0165f08dac3384594c18f1e76e8619ec81b816b157: Status 404 returned error can't find the container with id 3adfc47fa1103f29ce818a0165f08dac3384594c18f1e76e8619ec81b816b157 Oct 30 18:07:05.077648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:05.077609 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:07:05.077742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:05.077685 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:07:05.077742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:05.077717 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:07:05.083285 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:07:05.083239 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:07:05.786414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:05.786372 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" event={"ID":"357860ec-9d4e-4d5e-be19-eec58835b3dc","Type":"ContainerStarted","Data":"051b3f39ef48e1356b850dc0c578de9b542c6d9c588ae6556a252e8add3edfc3"} Oct 30 18:07:05.787179 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:05.787148 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" event={"ID":"357860ec-9d4e-4d5e-be19-eec58835b3dc","Type":"ContainerStarted","Data":"3adfc47fa1103f29ce818a0165f08dac3384594c18f1e76e8619ec81b816b157"} Oct 30 18:07:05.839539 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:05.839476 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podStartSLOduration=1.839459062 podStartE2EDuration="1.839459062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:05.837684006 +0000 UTC m=+884.794687842" watchObservedRunningTime="2026-10-30 18:07:05.839459062 +0000 UTC m=+884.796462900" Oct 30 18:07:06.741092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:06.741047 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:06.742960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:06.742917 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:07:06.743096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:06.742983 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:07:07.741628 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:07.741584 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:07:07.742045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:07.741646 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:07:08.742051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:08.742013 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:07:08.742434 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:08.742069 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:07:09.704213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:09.702837 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2"] Oct 30 18:07:09.707896 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:09.707867 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:09.722309 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:09.721957 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2"] Oct 30 18:07:09.758042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:09.750074 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:07:09.758042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:09.750136 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:07:09.758042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:09.750621 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3cc458\"/\"e2e-conflict-a-bc717f7f-kserve-self-signed-certs\"" Oct 30 18:07:09.899983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:09.893827 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-dshm\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:09.899983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:09.894147 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-home\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:09.899983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:09.894216 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-tmp-dir\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:09.899983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:09.894259 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-model-cache\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:09.899983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:09.894535 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b952edaf-d0ba-4679-a6ea-9ce101e17021-tls-certs\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:09.899983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:09.894581 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmdxd\" (UniqueName: \"kubernetes.io/projected/b952edaf-d0ba-4679-a6ea-9ce101e17021-kube-api-access-hmdxd\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.022860 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.022754 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-dshm\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.027034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.027000 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-home\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.027723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.027701 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-tmp-dir\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.028249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.028228 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-model-cache\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.028757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.028740 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b952edaf-d0ba-4679-a6ea-9ce101e17021-tls-certs\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.029360 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.029341 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmdxd\" (UniqueName: \"kubernetes.io/projected/b952edaf-d0ba-4679-a6ea-9ce101e17021-kube-api-access-hmdxd\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.042314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.028605 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-model-cache\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.042587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.028164 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-tmp-dir\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.042714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.027595 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-home\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.082429 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.082395 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b952edaf-d0ba-4679-a6ea-9ce101e17021-tls-certs\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.083281 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.083207 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmdxd\" (UniqueName: \"kubernetes.io/projected/b952edaf-d0ba-4679-a6ea-9ce101e17021-kube-api-access-hmdxd\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.083704 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.083685 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-dshm\") pod \"e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.232230 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.232186 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:10.643995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.634858 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2"] Oct 30 18:07:10.747095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.742339 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:07:10.747095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.742397 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:07:10.995964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:10.994728 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" event={"ID":"b952edaf-d0ba-4679-a6ea-9ce101e17021","Type":"ContainerStarted","Data":"b0f4865954fa3ffda6aefc42aba5ea51659c5bb9880accf869fe23e48fa9a96a"} Oct 30 18:07:11.744394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:11.744353 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:07:11.744696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:11.744664 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:07:11.757097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:11.757043 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:12.030991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:12.025705 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" event={"ID":"b952edaf-d0ba-4679-a6ea-9ce101e17021","Type":"ContainerStarted","Data":"d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5"} Oct 30 18:07:12.083106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:12.083029 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" podStartSLOduration=3.082994354 podStartE2EDuration="3.082994354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:12.081558829 +0000 UTC m=+891.038562667" watchObservedRunningTime="2026-10-30 18:07:12.082994354 +0000 UTC m=+891.039998195" Oct 30 18:07:12.233070 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:12.233030 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:12.620959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:12.620088 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:12.743959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:12.743334 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:07:12.743959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:12.743417 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:07:12.959020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:12.957129 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:13.174965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:13.168872 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:13.445957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:13.445090 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:07:13.445957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:13.445178 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:14.513461 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:14.513429 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:14.514363 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:14.514338 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:14.514996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:14.514966 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:07:20.233087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:20.233040 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:20.245918 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:20.245887 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:21.115189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:21.115156 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:07:21.304760 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:21.304729 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:07:21.309034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:21.309002 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:07:21.313051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:21.313027 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:07:21.317034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:21.317011 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:07:23.201584 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.201546 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k"] Oct 30 18:07:23.206573 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.206546 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:23.227464 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.227427 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k"] Oct 30 18:07:23.236808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.236773 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:07:23.236808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.236777 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:07:23.237091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.236835 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:07:23.237091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.237006 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:07:23.270287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.270211 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6cjl\" (UniqueName: \"kubernetes.io/projected/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-kube-api-access-x6cjl\") pod \"maas-api-revoke-sub-b2536922-bt27k\" (UID: \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:23.270470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.270413 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2536922-bt27k\" (UID: \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:23.270470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.270442 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2536922-bt27k\" (UID: \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:23.371041 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.371008 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2536922-bt27k\" (UID: \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:23.371214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.371059 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2536922-bt27k\" (UID: \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:23.371214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.371140 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6cjl\" (UniqueName: \"kubernetes.io/projected/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-kube-api-access-x6cjl\") pod \"maas-api-revoke-sub-b2536922-bt27k\" (UID: \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:23.372005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.371927 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2536922-bt27k\" (UID: \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:23.373804 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.373773 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2536922-bt27k\" (UID: \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:23.396335 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.396300 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6cjl\" (UniqueName: \"kubernetes.io/projected/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-kube-api-access-x6cjl\") pod \"maas-api-revoke-sub-b2536922-bt27k\" (UID: \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:23.519813 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.519721 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:23.826135 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:23.826094 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k"] Oct 30 18:07:24.123906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:24.123871 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" event={"ID":"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb","Type":"ContainerStarted","Data":"0f34642e5ab733cdec902511b272d48732d7cbce11cb9e0757288e95bec59e81"} Oct 30 18:07:25.132144 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:25.132050 4883 generic.go:356] "Generic (PLEG): container finished" podID="b4a8e400-b4f9-4d1b-ba69-73963dbc5adb" containerID="d9d9d0644b4badc85d465ce3b30fcb1233d84a0fa737fc04ea147b32c6aab198" exitCode=0 Oct 30 18:07:25.132536 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:25.132133 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" event={"ID":"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb","Type":"ContainerDied","Data":"d9d9d0644b4badc85d465ce3b30fcb1233d84a0fa737fc04ea147b32c6aab198"} Oct 30 18:07:26.294134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:26.294105 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:26.409549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:26.409464 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-openshift-ca\") pod \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\" (UID: \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\") " Oct 30 18:07:26.409549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:26.409531 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-xks-ca\") pod \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\" (UID: \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\") " Oct 30 18:07:26.409781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:26.409600 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-kube-api-access-x6cjl\" (UniqueName: \"kubernetes.io/projected/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-kube-api-access-x6cjl\") pod \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\" (UID: \"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb\") " Oct 30 18:07:26.409884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:26.409859 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-openshift-ca" pod "b4a8e400-b4f9-4d1b-ba69-73963dbc5adb" (UID: "b4a8e400-b4f9-4d1b-ba69-73963dbc5adb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:26.411740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:26.411703 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-kube-api-access-x6cjl" pod "b4a8e400-b4f9-4d1b-ba69-73963dbc5adb" (UID: "b4a8e400-b4f9-4d1b-ba69-73963dbc5adb"). InnerVolumeSpecName "kube-api-access-x6cjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:26.411963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:26.411944 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-xks-ca" pod "b4a8e400-b4f9-4d1b-ba69-73963dbc5adb" (UID: "b4a8e400-b4f9-4d1b-ba69-73963dbc5adb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:26.511602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:26.511563 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:07:26.511602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:26.511599 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6cjl\" (UniqueName: \"kubernetes.io/projected/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-kube-api-access-x6cjl\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:07:26.511602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:26.511611 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:07:27.145542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:27.145500 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" event={"ID":"b4a8e400-b4f9-4d1b-ba69-73963dbc5adb","Type":"ContainerDied","Data":"0f34642e5ab733cdec902511b272d48732d7cbce11cb9e0757288e95bec59e81"} Oct 30 18:07:27.145542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:27.145534 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k" Oct 30 18:07:27.145762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:27.145562 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f34642e5ab733cdec902511b272d48732d7cbce11cb9e0757288e95bec59e81" Oct 30 18:07:27.891641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:27.891602 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5"] Oct 30 18:07:27.892192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:27.892172 4883 cpu_manager.go:414] "containerMap: removing container" podUID="b4a8e400-b4f9-4d1b-ba69-73963dbc5adb" containerName="revoke-keys" Oct 30 18:07:27.892192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:27.892192 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a8e400-b4f9-4d1b-ba69-73963dbc5adb" containerName="revoke-keys" Oct 30 18:07:27.892353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:27.892309 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a8e400-b4f9-4d1b-ba69-73963dbc5adb" containerName="revoke-keys" Oct 30 18:07:27.896042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:27.896018 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:27.917296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:27.917262 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5"] Oct 30 18:07:27.926407 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:27.926372 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-57ee15\"/\"distinct-w0-57ee15-kserve-self-signed-certs\"" Oct 30 18:07:28.027956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.027907 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-dshm\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.028114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.028031 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btwg6\" (UniqueName: \"kubernetes.io/projected/50f45f5e-25c0-4132-88e1-92b762f1d334-kube-api-access-btwg6\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.028174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.028145 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-home\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.028226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.028203 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-tmp-dir\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.028261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.028241 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-model-cache\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.028297 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.028275 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f45f5e-25c0-4132-88e1-92b762f1d334-tls-certs\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.129409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.129365 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-dshm\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.130135 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.130107 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btwg6\" (UniqueName: \"kubernetes.io/projected/50f45f5e-25c0-4132-88e1-92b762f1d334-kube-api-access-btwg6\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.137960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.130634 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-home\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.139273 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.131097 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-home\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.139535 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.139500 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-tmp-dir\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.140086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.139958 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-tmp-dir\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.140490 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.140453 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-model-cache\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.140589 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.140031 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-model-cache\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.140713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.140700 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f45f5e-25c0-4132-88e1-92b762f1d334-tls-certs\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.152336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.149238 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-dshm\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.152336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.152128 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f45f5e-25c0-4132-88e1-92b762f1d334-tls-certs\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.156263 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.156208 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btwg6\" (UniqueName: \"kubernetes.io/projected/50f45f5e-25c0-4132-88e1-92b762f1d334-kube-api-access-btwg6\") pod \"distinct-w0-57ee15-kserve-545d7bc79c-jn6l5\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.210160 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.210125 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:28.522214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:28.522188 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5"] Oct 30 18:07:29.169564 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:29.169527 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" event={"ID":"50f45f5e-25c0-4132-88e1-92b762f1d334","Type":"ContainerStarted","Data":"dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1"} Oct 30 18:07:29.169564 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:29.169572 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" event={"ID":"50f45f5e-25c0-4132-88e1-92b762f1d334","Type":"ContainerStarted","Data":"b00e3a54fac1aed7639c22216637fd82ba66b44a16f3dc94cd7c6f5d789c7a56"} Oct 30 18:07:29.211059 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:29.211022 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:29.216214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:29.216154 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podStartSLOduration=2.21613653 podStartE2EDuration="2.21613653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:29.215083471 +0000 UTC m=+908.172087307" watchObservedRunningTime="2026-10-30 18:07:29.21613653 +0000 UTC m=+908.173140366" Oct 30 18:07:35.081917 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:07:35.081880 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:07:37.999470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:37.999430 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z"] Oct 30 18:07:38.004270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.004237 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.048410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.048366 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z"] Oct 30 18:07:38.048756 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.048731 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3cc458\"/\"e2e-conflict-b-bc717f7f-kserve-self-signed-certs\"" Oct 30 18:07:38.084645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.084595 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-dshm\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.084847 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.084690 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpzq5\" (UniqueName: \"kubernetes.io/projected/dbdbb2cc-d282-449f-9cda-78c927d87d39-kube-api-access-xpzq5\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.084847 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.084757 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-home\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.084996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.084844 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-model-cache\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.084996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.084880 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-tmp-dir\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.084996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.084960 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdbb2cc-d282-449f-9cda-78c927d87d39-tls-certs\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.185773 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.185735 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpzq5\" (UniqueName: \"kubernetes.io/projected/dbdbb2cc-d282-449f-9cda-78c927d87d39-kube-api-access-xpzq5\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.185992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.185798 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-home\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.185992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.185848 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-model-cache\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.185992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.185865 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-tmp-dir\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.185992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.185946 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdbb2cc-d282-449f-9cda-78c927d87d39-tls-certs\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.186219 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.186079 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-dshm\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.186458 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.186386 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-home\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.186458 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.186417 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-model-cache\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.186458 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.186447 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-tmp-dir\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.189426 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.189396 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdbb2cc-d282-449f-9cda-78c927d87d39-tls-certs\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.190153 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.190128 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-dshm\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.212833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.210479 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:38.249011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.248916 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpzq5\" (UniqueName: \"kubernetes.io/projected/dbdbb2cc-d282-449f-9cda-78c927d87d39-kube-api-access-xpzq5\") pod \"e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.338016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.333437 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:38.340999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.338879 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:38.361123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.361089 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:07:38.665741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:38.665704 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z"] Oct 30 18:07:38.666702 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:07:38.666671 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbdbb2cc_d282_449f_9cda_78c927d87d39.slice/crio-c24501e3a0d697d5402a48e50ff5a4d42de951487f1cf8eb993a0fdf343f99c7 WatchSource:0}: Error finding container c24501e3a0d697d5402a48e50ff5a4d42de951487f1cf8eb993a0fdf343f99c7: Status 404 returned error can't find the container with id c24501e3a0d697d5402a48e50ff5a4d42de951487f1cf8eb993a0fdf343f99c7 Oct 30 18:07:39.266827 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:39.266788 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" event={"ID":"dbdbb2cc-d282-449f-9cda-78c927d87d39","Type":"ContainerStarted","Data":"ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85"} Oct 30 18:07:39.267465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:39.267438 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" event={"ID":"dbdbb2cc-d282-449f-9cda-78c927d87d39","Type":"ContainerStarted","Data":"c24501e3a0d697d5402a48e50ff5a4d42de951487f1cf8eb993a0fdf343f99c7"} Oct 30 18:07:39.312803 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:39.310757 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" podStartSLOduration=2.310742961 podStartE2EDuration="2.310742961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:39.310381873 +0000 UTC m=+918.267385710" watchObservedRunningTime="2026-10-30 18:07:39.310742961 +0000 UTC m=+918.267746798" Oct 30 18:07:39.340022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:39.339982 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:48.340433 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:48.340386 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:48.354218 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:48.354178 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:07:49.357262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:07:49.357222 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:08:03.402752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:03.398882 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z"] Oct 30 18:08:04.585025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.584987 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk"] Oct 30 18:08:04.590775 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.590739 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.627010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.626969 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk"] Oct 30 18:08:04.656211 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.656166 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9hkg\" (UniqueName: \"kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-kube-api-access-q9hkg\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.656545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.656514 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.656673 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.656648 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.656759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.656743 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.656811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.656777 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.656863 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.656807 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.656863 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.656854 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.657530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.656886 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.657530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.656983 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.758283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.758247 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9hkg\" (UniqueName: \"kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-kube-api-access-q9hkg\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.758504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.758293 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.758504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.758392 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.758504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.758454 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.758504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.758481 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.758733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.758508 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.758733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.758545 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.758733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.758567 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.758733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.758595 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.758967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.758838 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.759033 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.758999 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-certs\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.759392 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.759364 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-credential-socket\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.759590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.759564 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istiod-ca-cert\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.759773 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.759755 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-data\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.761681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.761660 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-podinfo\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.762743 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.762720 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-envoy\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.787540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.787499 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9hkg\" (UniqueName: \"kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-kube-api-access-q9hkg\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.793822 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.793785 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-token\") pod \"e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:04.910705 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:04.910666 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:05.181314 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:08:05.181217 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:08:05.277282 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:05.276451 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk"] Oct 30 18:08:05.280203 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:08:05.280170 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9b738e_f6a3_40d6_b324_26efc6dfe7ea.slice/crio-c9b9f549498fb0fd1b6ebea95a020a16549fdaf72283fceeb9184c6ee9721994 WatchSource:0}: Error finding container c9b9f549498fb0fd1b6ebea95a020a16549fdaf72283fceeb9184c6ee9721994: Status 404 returned error can't find the container with id c9b9f549498fb0fd1b6ebea95a020a16549fdaf72283fceeb9184c6ee9721994 Oct 30 18:08:05.283326 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:05.283287 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:05.283450 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:05.283359 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:05.283450 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:05.283389 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:05.466758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:05.466643 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" event={"ID":"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea","Type":"ContainerStarted","Data":"45ed41507d39622999d6b7ddbea3fff5d193bf4bdebbf183916bfa2d73f04e8a"} Oct 30 18:08:05.466758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:05.466718 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" event={"ID":"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea","Type":"ContainerStarted","Data":"c9b9f549498fb0fd1b6ebea95a020a16549fdaf72283fceeb9184c6ee9721994"} Oct 30 18:08:05.521242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:05.521179 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podStartSLOduration=1.521163074 podStartE2EDuration="1.521163074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:05.517636849 +0000 UTC m=+944.474640685" watchObservedRunningTime="2026-10-30 18:08:05.521163074 +0000 UTC m=+944.478166979" Oct 30 18:08:05.911447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:05.911405 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:06.913480 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:06.913444 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:08:06.913983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:06.913502 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:08:07.911468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:07.911426 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:08:07.911655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:07.911490 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:08:08.021470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.021318 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl"] Oct 30 18:08:08.026189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.026164 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.043453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.043384 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl"] Oct 30 18:08:08.053242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.053203 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-57ee15\"/\"distinct-2-w0-57ee15-kserve-self-signed-certs\"" Oct 30 18:08:08.127816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.122622 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-dshm\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.127816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.122682 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-home\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.127816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.122778 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tmp-dir\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.127816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.122853 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-model-cache\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.127816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.122883 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2vtp\" (UniqueName: \"kubernetes.io/projected/f930967b-8f0f-48d5-921a-0d1c7b1bf289-kube-api-access-b2vtp\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.127816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.123116 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tls-certs\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.226232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.224020 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tls-certs\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.226232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.224200 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-dshm\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.226232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.224254 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-home\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.226232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.224301 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tmp-dir\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.226232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.224367 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-model-cache\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.226232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.224390 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2vtp\" (UniqueName: \"kubernetes.io/projected/f930967b-8f0f-48d5-921a-0d1c7b1bf289-kube-api-access-b2vtp\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.226232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.225482 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-home\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.237842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.237805 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tmp-dir\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.242969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.238341 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-model-cache\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.247044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.240645 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tls-certs\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.247271 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.242258 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-dshm\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.260125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.260080 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2vtp\" (UniqueName: \"kubernetes.io/projected/f930967b-8f0f-48d5-921a-0d1c7b1bf289-kube-api-access-b2vtp\") pod \"distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.341512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.341477 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:08.702606 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.702576 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl"] Oct 30 18:08:08.703686 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:08:08.703662 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf930967b_8f0f_48d5_921a_0d1c7b1bf289.slice/crio-eb133a1f311ac6152d0c4b7dc72de5232de003dc9a9f67f2f20beee10982a623 WatchSource:0}: Error finding container eb133a1f311ac6152d0c4b7dc72de5232de003dc9a9f67f2f20beee10982a623: Status 404 returned error can't find the container with id eb133a1f311ac6152d0c4b7dc72de5232de003dc9a9f67f2f20beee10982a623 Oct 30 18:08:08.911517 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.911476 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:08:08.911701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:08.911535 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:08:09.462741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.462703 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2"] Oct 30 18:08:09.467987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.466539 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.503171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.503135 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2"] Oct 30 18:08:09.524242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.524196 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" event={"ID":"f930967b-8f0f-48d5-921a-0d1c7b1bf289","Type":"ContainerStarted","Data":"baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42"} Oct 30 18:08:09.524513 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.524254 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" event={"ID":"f930967b-8f0f-48d5-921a-0d1c7b1bf289","Type":"ContainerStarted","Data":"eb133a1f311ac6152d0c4b7dc72de5232de003dc9a9f67f2f20beee10982a623"} Oct 30 18:08:09.543790 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.543740 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgrq5\" (UniqueName: \"kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-kube-api-access-dgrq5\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.543994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.543812 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.543994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.543964 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.544139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.544098 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.544222 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.544177 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.544222 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.544206 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.544367 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.544346 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.544434 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.544417 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c5b4960-719d-4f28-893f-a6d4a7e41369-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.544533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.544512 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.594266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.594197 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podStartSLOduration=2.594173223 podStartE2EDuration="2.594173223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:09.592672203 +0000 UTC m=+948.549676048" watchObservedRunningTime="2026-10-30 18:08:09.594173223 +0000 UTC m=+948.551177060" Oct 30 18:08:09.645520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.645482 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.645520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.645526 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.645792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.645597 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.645792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.645657 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c5b4960-719d-4f28-893f-a6d4a7e41369-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.645792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.645696 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.645984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.645792 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgrq5\" (UniqueName: \"kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-kube-api-access-dgrq5\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.645984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.645846 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.645984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.645873 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.645984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.645925 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.646191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.646046 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.646191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.646097 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.646302 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.646263 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.646643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.646616 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.646897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.646873 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c5b4960-719d-4f28-893f-a6d4a7e41369-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.648369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.648345 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.648896 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.648872 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.694961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.691317 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgrq5\" (UniqueName: \"kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-kube-api-access-dgrq5\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.694961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.694666 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.810013 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.808462 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:09.912703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.912174 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:08:09.912703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:09.912244 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:08:10.202065 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:10.196860 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2"] Oct 30 18:08:10.207019 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:08:10.206984 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5b4960_719d_4f28_893f_a6d4a7e41369.slice/crio-a696da2c73885292e6a36b92573a8469529615885d82169690aa7291d16bcca1 WatchSource:0}: Error finding container a696da2c73885292e6a36b92573a8469529615885d82169690aa7291d16bcca1: Status 404 returned error can't find the container with id a696da2c73885292e6a36b92573a8469529615885d82169690aa7291d16bcca1 Oct 30 18:08:10.210245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:10.210192 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:10.210355 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:10.210312 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:10.210414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:10.210362 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:10.342023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:10.341988 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:10.612974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:10.609930 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" event={"ID":"3c5b4960-719d-4f28-893f-a6d4a7e41369","Type":"ContainerStarted","Data":"f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956"} Oct 30 18:08:10.612974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:10.610189 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" event={"ID":"3c5b4960-719d-4f28-893f-a6d4a7e41369","Type":"ContainerStarted","Data":"a696da2c73885292e6a36b92573a8469529615885d82169690aa7291d16bcca1"} Oct 30 18:08:10.703865 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:10.703795 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podStartSLOduration=1.703771096 podStartE2EDuration="1.703771096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:10.675820226 +0000 UTC m=+949.632824064" watchObservedRunningTime="2026-10-30 18:08:10.703771096 +0000 UTC m=+949.660774933" Oct 30 18:08:10.812091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:10.812054 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:10.914446 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:10.914354 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:08:10.914731 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:10.914692 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:08:11.809963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:11.799766 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:11.828175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:11.828125 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:08:11.828394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:11.828201 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 18:08:11.917962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:11.911122 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:08:11.917962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:11.911196 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:08:12.119960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.119035 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" podUID="b952edaf-d0ba-4679-a6ea-9ce101e17021" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:12.358959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.357002 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" podUID="dbdbb2cc-d282-449f-9cda-78c927d87d39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:08:12.381681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.381578 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:12.628179 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.628112 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:08:12.685258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.685140 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:08:12.800085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.800032 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:12.801139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.801114 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:08:12.812412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.812374 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:08:12.813060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.813024 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 18:08:12.852665 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.852618 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:12.930963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.928822 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:08:12.952024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.928902 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:08:12.958285 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:12.953096 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:08:13.097958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.097282 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:13.181157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.125156 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" podUID="b952edaf-d0ba-4679-a6ea-9ce101e17021" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:13.181157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.125668 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:08:13.644789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.644716 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:08:13.836576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.645101 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:13.837226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.657032 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" podUID="dbdbb2cc-d282-449f-9cda-78c927d87d39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:08:13.837325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.748525 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:08:13.837412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.827619 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:08:13.837500 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.803437 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:08:13.837579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.802207 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:13.954962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.888669 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:08:13.954962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.924143 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:13.954962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.947071 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 18:08:13.971645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.971601 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:08:13.971984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.971964 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:08:13.972172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.972158 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:08:13.972322 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.972310 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:08:13.976131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:13.976103 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:08:14.027800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.027751 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:08:14.046473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.046409 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:08:14.046742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.046707 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:08:14.079081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.079040 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:08:14.100121 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.098031 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:14.100121 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.099465 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:08:14.134604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.134551 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" podUID="b952edaf-d0ba-4679-a6ea-9ce101e17021" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:14.328380 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.328336 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:08:14.351159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.350585 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:08:14.379316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.379230 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:08:14.380472 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.380433 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:08:14.380723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.380695 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:08:14.841993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.839552 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:14.841993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.840482 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:08:14.842657 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.842635 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:08:14.842807 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.842784 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 18:08:14.903004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.883451 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:08:14.903004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.883928 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:08:14.992683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:14.990410 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:15.151111 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:15.151033 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:08:15.276252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:15.272603 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:15.276252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:15.274134 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:15.286034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:15.276764 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:08:15.809612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:15.809575 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:08:15.809840 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:15.809631 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 18:08:16.270991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:16.261158 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2"] Oct 30 18:08:16.809460 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:16.809419 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:08:16.809654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:16.809490 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 18:08:17.809498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:17.809459 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:08:17.809913 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:17.809518 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 18:08:18.341920 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.341883 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:18.355380 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.355345 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:18.429161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.429112 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" podUID="dbdbb2cc-d282-449f-9cda-78c927d87d39" containerName="main" containerID="cri-o://ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85" gracePeriod=45 Oct 30 18:08:18.739729 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.739699 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:08:18.801668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.801621 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbdbb2cc-d282-449f-9cda-78c927d87d39-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdbb2cc-d282-449f-9cda-78c927d87d39-tls-certs\") pod \"dbdbb2cc-d282-449f-9cda-78c927d87d39\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " Oct 30 18:08:18.801875 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.801719 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-dshm\") pod \"dbdbb2cc-d282-449f-9cda-78c927d87d39\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " Oct 30 18:08:18.801875 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.801785 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbdbb2cc-d282-449f-9cda-78c927d87d39-kube-api-access-xpzq5\" (UniqueName: \"kubernetes.io/projected/dbdbb2cc-d282-449f-9cda-78c927d87d39-kube-api-access-xpzq5\") pod \"dbdbb2cc-d282-449f-9cda-78c927d87d39\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " Oct 30 18:08:18.802030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.801894 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-model-cache\") pod \"dbdbb2cc-d282-449f-9cda-78c927d87d39\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " Oct 30 18:08:18.802030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.802023 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-home\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-home\") pod \"dbdbb2cc-d282-449f-9cda-78c927d87d39\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " Oct 30 18:08:18.802353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.802325 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-home" pod "dbdbb2cc-d282-449f-9cda-78c927d87d39" (UID: "dbdbb2cc-d282-449f-9cda-78c927d87d39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:18.802481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.802403 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-model-cache" pod "dbdbb2cc-d282-449f-9cda-78c927d87d39" (UID: "dbdbb2cc-d282-449f-9cda-78c927d87d39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:18.802552 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.802484 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-tmp-dir\") pod \"dbdbb2cc-d282-449f-9cda-78c927d87d39\" (UID: \"dbdbb2cc-d282-449f-9cda-78c927d87d39\") " Oct 30 18:08:18.802770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.802739 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-tmp-dir" pod "dbdbb2cc-d282-449f-9cda-78c927d87d39" (UID: "dbdbb2cc-d282-449f-9cda-78c927d87d39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:18.803092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.803071 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:18.803194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.803096 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:18.803194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.803109 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:18.805023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.804988 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdbb2cc-d282-449f-9cda-78c927d87d39-kube-api-access-xpzq5" pod "dbdbb2cc-d282-449f-9cda-78c927d87d39" (UID: "dbdbb2cc-d282-449f-9cda-78c927d87d39"). InnerVolumeSpecName "kube-api-access-xpzq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:18.805616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.805590 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbdbb2cc-d282-449f-9cda-78c927d87d39-tls-certs" pod "dbdbb2cc-d282-449f-9cda-78c927d87d39" (UID: "dbdbb2cc-d282-449f-9cda-78c927d87d39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:18.805955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.805912 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-dshm" pod "dbdbb2cc-d282-449f-9cda-78c927d87d39" (UID: "dbdbb2cc-d282-449f-9cda-78c927d87d39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:18.809284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.809256 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:08:18.809389 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.809311 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 18:08:18.904492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.904402 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpzq5\" (UniqueName: \"kubernetes.io/projected/dbdbb2cc-d282-449f-9cda-78c927d87d39-kube-api-access-xpzq5\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:18.904492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.904439 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdbb2cc-d282-449f-9cda-78c927d87d39-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:18.904492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:18.904450 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdbb2cc-d282-449f-9cda-78c927d87d39-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:19.131795 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:19.131755 4883 generic.go:356] "Generic (PLEG): container finished" podID="dbdbb2cc-d282-449f-9cda-78c927d87d39" containerID="ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85" exitCode=0 Oct 30 18:08:19.131795 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:19.131804 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" Oct 30 18:08:19.132042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:19.131839 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" event={"ID":"dbdbb2cc-d282-449f-9cda-78c927d87d39","Type":"ContainerDied","Data":"ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85"} Oct 30 18:08:19.132042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:19.131889 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z" event={"ID":"dbdbb2cc-d282-449f-9cda-78c927d87d39","Type":"ContainerDied","Data":"c24501e3a0d697d5402a48e50ff5a4d42de951487f1cf8eb993a0fdf343f99c7"} Oct 30 18:08:19.132042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:19.131910 4883 scope.go:122] "RemoveContainer" containerID="ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85" Oct 30 18:08:19.144522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:19.144500 4883 scope.go:122] "RemoveContainer" containerID="ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85" Oct 30 18:08:19.144891 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:08:19.144867 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85\": container with ID starting with ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85 not found: ID does not exist" containerID="ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85" Oct 30 18:08:19.145029 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:19.144897 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85"} err="failed to get container status \"ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85\": rpc error: code = NotFound desc = could not find container \"ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85\": container with ID starting with ea469d0d2eff65c81cd2f6f0e552c12d49436f60ed8931075d5e93feb0fa2b85 not found: ID does not exist" Oct 30 18:08:19.147122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:19.147098 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:08:19.231855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:19.231757 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z"] Oct 30 18:08:19.237903 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:19.237867 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/e2e-conflict-b-bc717f7f-kserve-57655c8fcb-ftz9z"] Oct 30 18:08:19.809210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:19.809172 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:08:19.809404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:19.809230 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 18:08:21.179476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:21.179439 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbdbb2cc-d282-449f-9cda-78c927d87d39" path="/var/lib/kubelet/pods/dbdbb2cc-d282-449f-9cda-78c927d87d39/volumes" Oct 30 18:08:21.809836 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:21.809803 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:21.810312 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:21.810287 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:21.811433 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:21.811410 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:08:24.747765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.747721 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw"] Oct 30 18:08:24.748344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.748314 4883 cpu_manager.go:414] "containerMap: removing container" podUID="dbdbb2cc-d282-449f-9cda-78c927d87d39" containerName="main" Oct 30 18:08:24.748344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.748333 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbdbb2cc-d282-449f-9cda-78c927d87d39" containerName="main" Oct 30 18:08:24.748540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.748418 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbdbb2cc-d282-449f-9cda-78c927d87d39" containerName="main" Oct 30 18:08:24.753499 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.753469 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.779220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.779180 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw"] Oct 30 18:08:24.798026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.797914 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz"] Oct 30 18:08:24.803782 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.803199 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.824820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.824777 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz"] Oct 30 18:08:24.888202 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888157 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.888364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888207 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.888364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888280 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.888364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888320 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.888364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888343 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.888525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888420 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdchz\" (UniqueName: \"kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-kube-api-access-wdchz\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.888525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888492 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.888525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888518 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.888686 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888588 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.888686 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888639 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.888686 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888665 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.888796 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888733 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.888796 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888787 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.888866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888847 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.888901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888873 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wchh\" (UniqueName: \"kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-kube-api-access-4wchh\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.888974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888911 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.889014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.888989 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.889053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.889018 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.990653 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.990609 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.990853 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.990664 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.990853 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.990692 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.990853 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.990734 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.990853 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.990793 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.990853 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.990830 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.991193 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.990863 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wchh\" (UniqueName: \"kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-kube-api-access-4wchh\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.991193 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.990922 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.991319 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991210 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.991319 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991270 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.991434 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991391 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.991546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991430 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.991546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991431 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-credential-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.991546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991523 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.991746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991570 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.991746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991603 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.991746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991634 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.991746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991663 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.991746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991709 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.991746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991721 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdchz\" (UniqueName: \"kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-kube-api-access-wdchz\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.992265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991793 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.992265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.991820 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.992265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.992188 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.992472 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.992328 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.992663 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.992638 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-socket\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.992663 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.992654 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-data\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.992870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.992838 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istiod-ca-cert\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.992870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.992870 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-certs\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.993804 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.993768 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:24.995118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.995090 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-envoy\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.995434 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.995411 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:24.995517 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:24.995481 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-podinfo\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:25.016846 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.016761 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:25.017088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.017059 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdchz\" (UniqueName: \"kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-kube-api-access-wdchz\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:25.017311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.017270 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-token\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:25.017811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.017785 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wchh\" (UniqueName: \"kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-kube-api-access-4wchh\") pod \"e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:25.068491 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.068443 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:25.124128 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.124086 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:25.402503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.402473 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw"] Oct 30 18:08:25.403788 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:08:25.403760 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edf8fc2_6fba_401b_8f7e_6b73830e1745.slice/crio-7009f008f11ae33aee8b70e5318f1a9309962cd33ea662b11f2a754cfcf4554e WatchSource:0}: Error finding container 7009f008f11ae33aee8b70e5318f1a9309962cd33ea662b11f2a754cfcf4554e: Status 404 returned error can't find the container with id 7009f008f11ae33aee8b70e5318f1a9309962cd33ea662b11f2a754cfcf4554e Oct 30 18:08:25.406477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.406438 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:25.406583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.406508 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:25.406583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.406537 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:25.466911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.466879 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz"] Oct 30 18:08:25.468180 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:08:25.468151 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349475cf_93a4_49d8_8f2f_61bb0b5b7bc0.slice/crio-bae0d150331485f1a338d041fd9596c96a8814bd44fee2325e16a5296a972f13 WatchSource:0}: Error finding container bae0d150331485f1a338d041fd9596c96a8814bd44fee2325e16a5296a972f13: Status 404 returned error can't find the container with id bae0d150331485f1a338d041fd9596c96a8814bd44fee2325e16a5296a972f13 Oct 30 18:08:25.470798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.470691 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:25.470798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.470756 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:25.470798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:25.470786 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:26.230158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:26.230103 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" event={"ID":"7edf8fc2-6fba-401b-8f7e-6b73830e1745","Type":"ContainerStarted","Data":"00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0"} Oct 30 18:08:26.230708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:26.230664 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" event={"ID":"7edf8fc2-6fba-401b-8f7e-6b73830e1745","Type":"ContainerStarted","Data":"7009f008f11ae33aee8b70e5318f1a9309962cd33ea662b11f2a754cfcf4554e"} Oct 30 18:08:26.232090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:26.232062 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" event={"ID":"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0","Type":"ContainerStarted","Data":"4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a"} Oct 30 18:08:26.232191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:26.232106 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" event={"ID":"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0","Type":"ContainerStarted","Data":"bae0d150331485f1a338d041fd9596c96a8814bd44fee2325e16a5296a972f13"} Oct 30 18:08:26.306159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:26.298624 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" podStartSLOduration=2.298602438 podStartE2EDuration="2.298602438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:26.297416533 +0000 UTC m=+965.254420394" watchObservedRunningTime="2026-10-30 18:08:26.298602438 +0000 UTC m=+965.255606276" Oct 30 18:08:26.356431 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:26.356108 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" podStartSLOduration=2.356090101 podStartE2EDuration="2.356090101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:26.35581225 +0000 UTC m=+965.312816086" watchObservedRunningTime="2026-10-30 18:08:26.356090101 +0000 UTC m=+965.313093939" Oct 30 18:08:27.076999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:27.073299 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:27.112268 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:27.112218 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:08:27.112483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:27.112293 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 18:08:27.125162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:27.125131 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:27.126482 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:27.126453 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:08:27.126621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:27.126511 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:08:28.071956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:28.071350 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:08:28.071956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:28.071423 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 18:08:28.125381 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:28.125343 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:08:28.125571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:28.125401 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:08:29.077958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:29.072386 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:08:29.077958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:29.072492 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 18:08:29.124766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:29.124730 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:08:29.124924 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:29.124788 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:08:30.082832 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:30.082786 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:08:30.083470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:30.083439 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 18:08:30.125323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:30.125290 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:08:30.125564 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:30.125524 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:08:31.083389 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.083351 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:08:31.084021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.083990 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 18:08:31.124494 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.124456 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:08:31.124672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.124516 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:08:31.348068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.347990 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" podUID="b952edaf-d0ba-4679-a6ea-9ce101e17021" containerName="main" containerID="cri-o://d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5" gracePeriod=45 Oct 30 18:08:31.754145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.754100 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f"] Oct 30 18:08:31.781665 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.763489 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.837749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.837706 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.839435 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.838006 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde015c2-b427-435f-87bf-935a6056e8b9-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.839771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.839741 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.840040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.840019 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clbsv\" (UniqueName: \"kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-kube-api-access-clbsv\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.840172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.840156 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.840282 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.840265 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde015c2-b427-435f-87bf-935a6056e8b9-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.840417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.840366 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.840533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.840519 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.840649 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.840634 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.841087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.839066 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f"] Oct 30 18:08:31.850172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.850130 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:08:31.941759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.941716 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-home\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-home\") pod \"b952edaf-d0ba-4679-a6ea-9ce101e17021\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " Oct 30 18:08:31.942083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.942059 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b952edaf-d0ba-4679-a6ea-9ce101e17021-tls-certs\" (UniqueName: \"kubernetes.io/secret/b952edaf-d0ba-4679-a6ea-9ce101e17021-tls-certs\") pod \"b952edaf-d0ba-4679-a6ea-9ce101e17021\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " Oct 30 18:08:31.942275 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.942257 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b952edaf-d0ba-4679-a6ea-9ce101e17021-kube-api-access-hmdxd\" (UniqueName: \"kubernetes.io/projected/b952edaf-d0ba-4679-a6ea-9ce101e17021-kube-api-access-hmdxd\") pod \"b952edaf-d0ba-4679-a6ea-9ce101e17021\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " Oct 30 18:08:31.942396 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.942382 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-dshm\") pod \"b952edaf-d0ba-4679-a6ea-9ce101e17021\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " Oct 30 18:08:31.942608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.942590 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-model-cache\") pod \"b952edaf-d0ba-4679-a6ea-9ce101e17021\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " Oct 30 18:08:31.942772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.942755 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-tmp-dir\") pod \"b952edaf-d0ba-4679-a6ea-9ce101e17021\" (UID: \"b952edaf-d0ba-4679-a6ea-9ce101e17021\") " Oct 30 18:08:31.943236 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.943207 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.943385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.943366 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clbsv\" (UniqueName: \"kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-kube-api-access-clbsv\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.943496 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.943482 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.943592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.943579 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde015c2-b427-435f-87bf-935a6056e8b9-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.943759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.943743 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.943870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.943855 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.944014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.943997 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.944372 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.944349 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.944527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.944501 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.944608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.944516 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde015c2-b427-435f-87bf-935a6056e8b9-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.945329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.945299 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-credential-socket\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.945721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.945700 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde015c2-b427-435f-87bf-935a6056e8b9-istiod-ca-cert\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.945836 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.945726 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-data\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.945954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.943888 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-model-cache" pod "b952edaf-d0ba-4679-a6ea-9ce101e17021" (UID: "b952edaf-d0ba-4679-a6ea-9ce101e17021"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:31.948454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.944133 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-home" pod "b952edaf-d0ba-4679-a6ea-9ce101e17021" (UID: "b952edaf-d0ba-4679-a6ea-9ce101e17021"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:31.948653 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.946570 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-certs\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.948757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.944714 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-tmp-dir" pod "b952edaf-d0ba-4679-a6ea-9ce101e17021" (UID: "b952edaf-d0ba-4679-a6ea-9ce101e17021"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:31.949956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.949909 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b952edaf-d0ba-4679-a6ea-9ce101e17021-tls-certs" pod "b952edaf-d0ba-4679-a6ea-9ce101e17021" (UID: "b952edaf-d0ba-4679-a6ea-9ce101e17021"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:31.950195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.950165 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b952edaf-d0ba-4679-a6ea-9ce101e17021-kube-api-access-hmdxd" pod "b952edaf-d0ba-4679-a6ea-9ce101e17021" (UID: "b952edaf-d0ba-4679-a6ea-9ce101e17021"). InnerVolumeSpecName "kube-api-access-hmdxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:31.951507 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.951454 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-envoy\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.951655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.951624 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-dshm" pod "b952edaf-d0ba-4679-a6ea-9ce101e17021" (UID: "b952edaf-d0ba-4679-a6ea-9ce101e17021"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:31.952614 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.952588 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde015c2-b427-435f-87bf-935a6056e8b9-istio-podinfo\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.974066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.973972 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clbsv\" (UniqueName: \"kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-kube-api-access-clbsv\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:31.976550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:31.976510 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-istio-token\") pod \"e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:32.046481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.046384 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:32.046481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.046419 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b952edaf-d0ba-4679-a6ea-9ce101e17021-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:32.046481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.046430 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmdxd\" (UniqueName: \"kubernetes.io/projected/b952edaf-d0ba-4679-a6ea-9ce101e17021-kube-api-access-hmdxd\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:32.046481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.046439 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:32.046481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.046447 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:32.046481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.046455 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b952edaf-d0ba-4679-a6ea-9ce101e17021-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:32.069388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.069345 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:08:32.069545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.069413 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 18:08:32.125182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.125137 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:08:32.125671 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.125208 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:08:32.194586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.194545 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:32.309397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.309297 4883 generic.go:356] "Generic (PLEG): container finished" podID="b952edaf-d0ba-4679-a6ea-9ce101e17021" containerID="d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5" exitCode=0 Oct 30 18:08:32.309547 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.309432 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" event={"ID":"b952edaf-d0ba-4679-a6ea-9ce101e17021","Type":"ContainerDied","Data":"d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5"} Oct 30 18:08:32.309547 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.309471 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" event={"ID":"b952edaf-d0ba-4679-a6ea-9ce101e17021","Type":"ContainerDied","Data":"b0f4865954fa3ffda6aefc42aba5ea51659c5bb9880accf869fe23e48fa9a96a"} Oct 30 18:08:32.309547 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.309495 4883 scope.go:122] "RemoveContainer" containerID="d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5" Oct 30 18:08:32.309685 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.309670 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2" Oct 30 18:08:32.335487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.335459 4883 scope.go:122] "RemoveContainer" containerID="d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5" Oct 30 18:08:32.335902 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:08:32.335877 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5\": container with ID starting with d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5 not found: ID does not exist" containerID="d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5" Oct 30 18:08:32.336064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.335913 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5"} err="failed to get container status \"d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5\": rpc error: code = NotFound desc = could not find container \"d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5\": container with ID starting with d47f9a2e559ce2341ff52316f7ce15a8d29196a2da4e29d5444262fd93871ec5 not found: ID does not exist" Oct 30 18:08:32.376835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.376581 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2"] Oct 30 18:08:32.383362 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.383322 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/e2e-conflict-a-bc717f7f-kserve-79784fb6f8-r76z2"] Oct 30 18:08:32.527716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.527669 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f"] Oct 30 18:08:32.534300 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:08:32.534267 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde015c2_b427_435f_87bf_935a6056e8b9.slice/crio-781495c5b1b865502e072c138569f3deb2a3e89c3fdbd2e879ea6b4402c6d378 WatchSource:0}: Error finding container 781495c5b1b865502e072c138569f3deb2a3e89c3fdbd2e879ea6b4402c6d378: Status 404 returned error can't find the container with id 781495c5b1b865502e072c138569f3deb2a3e89c3fdbd2e879ea6b4402c6d378 Oct 30 18:08:32.537268 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.537219 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:32.537359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.537326 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:32.537410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:32.537398 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:08:33.192960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:33.191771 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b952edaf-d0ba-4679-a6ea-9ce101e17021" path="/var/lib/kubelet/pods/b952edaf-d0ba-4679-a6ea-9ce101e17021/volumes" Oct 30 18:08:33.427135 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:33.427094 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" event={"ID":"cde015c2-b427-435f-87bf-935a6056e8b9","Type":"ContainerStarted","Data":"105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2"} Oct 30 18:08:33.427418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:33.427401 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" event={"ID":"cde015c2-b427-435f-87bf-935a6056e8b9","Type":"ContainerStarted","Data":"781495c5b1b865502e072c138569f3deb2a3e89c3fdbd2e879ea6b4402c6d378"} Oct 30 18:08:33.511542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:33.511396 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" podStartSLOduration=2.51137125 podStartE2EDuration="2.51137125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:33.489421041 +0000 UTC m=+972.446424900" watchObservedRunningTime="2026-10-30 18:08:33.51137125 +0000 UTC m=+972.468375087" Oct 30 18:08:33.720270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:33.720230 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:33.720746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:33.720719 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:33.722278 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:33.722236 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:08:33.807015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:33.804268 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:33.807015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:33.805521 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:33.816988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:33.813669 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:08:34.195538 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:34.195502 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:34.196901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:34.196874 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:08:34.197067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:34.196975 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:08:35.107073 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:08:35.107033 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:08:35.119165 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.119128 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn"] Oct 30 18:08:35.119707 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.119690 4883 cpu_manager.go:414] "containerMap: removing container" podUID="b952edaf-d0ba-4679-a6ea-9ce101e17021" containerName="main" Oct 30 18:08:35.119798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.119708 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b952edaf-d0ba-4679-a6ea-9ce101e17021" containerName="main" Oct 30 18:08:35.119798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.119792 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="b952edaf-d0ba-4679-a6ea-9ce101e17021" containerName="main" Oct 30 18:08:35.124533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.124512 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:35.145361 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.145325 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn"] Oct 30 18:08:35.154417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.154380 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:08:35.154695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.154670 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:08:35.154805 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.154756 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:08:35.154805 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.154721 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:08:35.195067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.194986 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:08:35.195067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.195050 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:08:35.227669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.227626 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n8dv\" (UniqueName: \"kubernetes.io/projected/49c78665-d28d-46c8-9e17-d5f4b4943319-kube-api-access-6n8dv\") pod \"maas-api-revoke-sub-95fe591e-kv7dn\" (UID: \"49c78665-d28d-46c8-9e17-d5f4b4943319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:35.228143 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.227855 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95fe591e-kv7dn\" (UID: \"49c78665-d28d-46c8-9e17-d5f4b4943319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:35.228143 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.227890 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95fe591e-kv7dn\" (UID: \"49c78665-d28d-46c8-9e17-d5f4b4943319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:35.329288 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.329245 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95fe591e-kv7dn\" (UID: \"49c78665-d28d-46c8-9e17-d5f4b4943319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:35.329288 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.329291 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95fe591e-kv7dn\" (UID: \"49c78665-d28d-46c8-9e17-d5f4b4943319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:35.329561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.329341 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n8dv\" (UniqueName: \"kubernetes.io/projected/49c78665-d28d-46c8-9e17-d5f4b4943319-kube-api-access-6n8dv\") pod \"maas-api-revoke-sub-95fe591e-kv7dn\" (UID: \"49c78665-d28d-46c8-9e17-d5f4b4943319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:35.330077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.330050 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95fe591e-kv7dn\" (UID: \"49c78665-d28d-46c8-9e17-d5f4b4943319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:35.331752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.331728 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95fe591e-kv7dn\" (UID: \"49c78665-d28d-46c8-9e17-d5f4b4943319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:35.354310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.354278 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n8dv\" (UniqueName: \"kubernetes.io/projected/49c78665-d28d-46c8-9e17-d5f4b4943319-kube-api-access-6n8dv\") pod \"maas-api-revoke-sub-95fe591e-kv7dn\" (UID: \"49c78665-d28d-46c8-9e17-d5f4b4943319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:35.438882 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.438782 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:35.792192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:35.792162 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn"] Oct 30 18:08:35.793800 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:08:35.793761 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c78665_d28d_46c8_9e17_d5f4b4943319.slice/crio-dad4d946540f6c466a4b8e2a6e2db6cc305586ddadba96165536143f09be1eca WatchSource:0}: Error finding container dad4d946540f6c466a4b8e2a6e2db6cc305586ddadba96165536143f09be1eca: Status 404 returned error can't find the container with id dad4d946540f6c466a4b8e2a6e2db6cc305586ddadba96165536143f09be1eca Oct 30 18:08:36.195585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:36.195548 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:08:36.196316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:36.195800 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:08:36.471024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:36.469699 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" event={"ID":"49c78665-d28d-46c8-9e17-d5f4b4943319","Type":"ContainerStarted","Data":"e8135e5d76b379e82508cb2d4055c57a19d974d1cffb2ffb1587e2583f933f51"} Oct 30 18:08:36.471024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:36.469772 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" event={"ID":"49c78665-d28d-46c8-9e17-d5f4b4943319","Type":"ContainerStarted","Data":"dad4d946540f6c466a4b8e2a6e2db6cc305586ddadba96165536143f09be1eca"} Oct 30 18:08:36.518135 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:36.518078 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" podStartSLOduration=1.518061308 podStartE2EDuration="1.518061308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:36.515056123 +0000 UTC m=+975.472059961" watchObservedRunningTime="2026-10-30 18:08:36.518061308 +0000 UTC m=+975.475065144" Oct 30 18:08:37.196960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:37.195901 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:08:37.196960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:37.195986 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:08:37.503962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:37.488520 4883 generic.go:356] "Generic (PLEG): container finished" podID="49c78665-d28d-46c8-9e17-d5f4b4943319" containerID="e8135e5d76b379e82508cb2d4055c57a19d974d1cffb2ffb1587e2583f933f51" exitCode=0 Oct 30 18:08:37.503962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:37.488715 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" event={"ID":"49c78665-d28d-46c8-9e17-d5f4b4943319","Type":"ContainerDied","Data":"e8135e5d76b379e82508cb2d4055c57a19d974d1cffb2ffb1587e2583f933f51"} Oct 30 18:08:38.195443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:38.195405 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:08:38.195608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:38.195460 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:08:38.736294 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:38.736266 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:38.834890 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:38.834802 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c78665-d28d-46c8-9e17-d5f4b4943319-kube-api-access-6n8dv\" (UniqueName: \"kubernetes.io/projected/49c78665-d28d-46c8-9e17-d5f4b4943319-kube-api-access-6n8dv\") pod \"49c78665-d28d-46c8-9e17-d5f4b4943319\" (UID: \"49c78665-d28d-46c8-9e17-d5f4b4943319\") " Oct 30 18:08:38.835078 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:38.834964 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-xks-ca\") pod \"49c78665-d28d-46c8-9e17-d5f4b4943319\" (UID: \"49c78665-d28d-46c8-9e17-d5f4b4943319\") " Oct 30 18:08:38.835147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:38.835114 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-openshift-ca\") pod \"49c78665-d28d-46c8-9e17-d5f4b4943319\" (UID: \"49c78665-d28d-46c8-9e17-d5f4b4943319\") " Oct 30 18:08:38.835702 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:38.835672 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-openshift-ca" pod "49c78665-d28d-46c8-9e17-d5f4b4943319" (UID: "49c78665-d28d-46c8-9e17-d5f4b4943319"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:38.837306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:38.837265 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c78665-d28d-46c8-9e17-d5f4b4943319-kube-api-access-6n8dv" pod "49c78665-d28d-46c8-9e17-d5f4b4943319" (UID: "49c78665-d28d-46c8-9e17-d5f4b4943319"). InnerVolumeSpecName "kube-api-access-6n8dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:38.837801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:38.837776 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-xks-ca" pod "49c78665-d28d-46c8-9e17-d5f4b4943319" (UID: "49c78665-d28d-46c8-9e17-d5f4b4943319"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:38.937067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:38.937030 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:38.937067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:38.937063 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c78665-d28d-46c8-9e17-d5f4b4943319-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:38.937067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:38.937077 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n8dv\" (UniqueName: \"kubernetes.io/projected/49c78665-d28d-46c8-9e17-d5f4b4943319-kube-api-access-6n8dv\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:39.195860 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:39.195826 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:08:39.196068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:39.195880 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 18:08:39.505981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:39.505838 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" Oct 30 18:08:39.505981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:39.505849 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn" event={"ID":"49c78665-d28d-46c8-9e17-d5f4b4943319","Type":"ContainerDied","Data":"dad4d946540f6c466a4b8e2a6e2db6cc305586ddadba96165536143f09be1eca"} Oct 30 18:08:39.505981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:39.505910 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dad4d946540f6c466a4b8e2a6e2db6cc305586ddadba96165536143f09be1eca" Oct 30 18:08:40.200714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:40.200682 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:40.201213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:40.200968 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:40.201742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:40.201725 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:08:50.003467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.003429 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz"] Oct 30 18:08:50.004038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.004021 4883 cpu_manager.go:414] "containerMap: removing container" podUID="49c78665-d28d-46c8-9e17-d5f4b4943319" containerName="revoke-keys" Oct 30 18:08:50.004091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.004040 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c78665-d28d-46c8-9e17-d5f4b4943319" containerName="revoke-keys" Oct 30 18:08:50.004143 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.004135 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c78665-d28d-46c8-9e17-d5f4b4943319" containerName="revoke-keys" Oct 30 18:08:50.009127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.009101 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:50.026075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.026028 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:08:50.029555 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.029514 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz"] Oct 30 18:08:50.038398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.038342 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:08:50.039398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.039373 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:08:50.039633 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.039599 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:08:50.075152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.075113 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbxs\" (UniqueName: \"kubernetes.io/projected/8a8ecd72-c079-440b-811a-903809ad700e-kube-api-access-mgbxs\") pod \"maas-api-revoke-sub-282922cb-fcnjz\" (UID: \"8a8ecd72-c079-440b-811a-903809ad700e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:50.075368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.075178 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-282922cb-fcnjz\" (UID: \"8a8ecd72-c079-440b-811a-903809ad700e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:50.075517 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.075491 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-282922cb-fcnjz\" (UID: \"8a8ecd72-c079-440b-811a-903809ad700e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:50.176840 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.176799 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-282922cb-fcnjz\" (UID: \"8a8ecd72-c079-440b-811a-903809ad700e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:50.177063 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.176877 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbxs\" (UniqueName: \"kubernetes.io/projected/8a8ecd72-c079-440b-811a-903809ad700e-kube-api-access-mgbxs\") pod \"maas-api-revoke-sub-282922cb-fcnjz\" (UID: \"8a8ecd72-c079-440b-811a-903809ad700e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:50.177063 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.176907 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-282922cb-fcnjz\" (UID: \"8a8ecd72-c079-440b-811a-903809ad700e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:50.177601 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.177578 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-282922cb-fcnjz\" (UID: \"8a8ecd72-c079-440b-811a-903809ad700e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:50.179421 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.179392 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-282922cb-fcnjz\" (UID: \"8a8ecd72-c079-440b-811a-903809ad700e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:50.201600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.201561 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbxs\" (UniqueName: \"kubernetes.io/projected/8a8ecd72-c079-440b-811a-903809ad700e-kube-api-access-mgbxs\") pod \"maas-api-revoke-sub-282922cb-fcnjz\" (UID: \"8a8ecd72-c079-440b-811a-903809ad700e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:50.324220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.324126 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:50.636172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:50.636134 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz"] Oct 30 18:08:50.637712 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:08:50.637677 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8ecd72_c079_440b_811a_903809ad700e.slice/crio-83e6dd0248fbecba17ec8da6128e70091574ff64575dec9c9a646353953d5389 WatchSource:0}: Error finding container 83e6dd0248fbecba17ec8da6128e70091574ff64575dec9c9a646353953d5389: Status 404 returned error can't find the container with id 83e6dd0248fbecba17ec8da6128e70091574ff64575dec9c9a646353953d5389 Oct 30 18:08:51.596205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:51.596165 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" event={"ID":"8a8ecd72-c079-440b-811a-903809ad700e","Type":"ContainerStarted","Data":"ba0750e925694fa9af6d14b5983309f33fefc24e0558fd0013754caa38efbe0e"} Oct 30 18:08:51.596641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:51.596231 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" event={"ID":"8a8ecd72-c079-440b-811a-903809ad700e","Type":"ContainerStarted","Data":"83e6dd0248fbecba17ec8da6128e70091574ff64575dec9c9a646353953d5389"} Oct 30 18:08:51.643489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:51.643426 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" podStartSLOduration=2.643409298 podStartE2EDuration="2.643409298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:51.639376671 +0000 UTC m=+990.596380586" watchObservedRunningTime="2026-10-30 18:08:51.643409298 +0000 UTC m=+990.600413134" Oct 30 18:08:52.604630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:52.604594 4883 generic.go:356] "Generic (PLEG): container finished" podID="8a8ecd72-c079-440b-811a-903809ad700e" containerID="ba0750e925694fa9af6d14b5983309f33fefc24e0558fd0013754caa38efbe0e" exitCode=0 Oct 30 18:08:52.605082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:52.604670 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" event={"ID":"8a8ecd72-c079-440b-811a-903809ad700e","Type":"ContainerDied","Data":"ba0750e925694fa9af6d14b5983309f33fefc24e0558fd0013754caa38efbe0e"} Oct 30 18:08:53.773897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:53.773857 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:53.825418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:53.825381 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-openshift-ca\") pod \"8a8ecd72-c079-440b-811a-903809ad700e\" (UID: \"8a8ecd72-c079-440b-811a-903809ad700e\") " Oct 30 18:08:53.825626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:53.825576 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8ecd72-c079-440b-811a-903809ad700e-kube-api-access-mgbxs\" (UniqueName: \"kubernetes.io/projected/8a8ecd72-c079-440b-811a-903809ad700e-kube-api-access-mgbxs\") pod \"8a8ecd72-c079-440b-811a-903809ad700e\" (UID: \"8a8ecd72-c079-440b-811a-903809ad700e\") " Oct 30 18:08:53.825626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:53.825606 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-xks-ca\") pod \"8a8ecd72-c079-440b-811a-903809ad700e\" (UID: \"8a8ecd72-c079-440b-811a-903809ad700e\") " Oct 30 18:08:53.826033 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:53.826004 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-openshift-ca" pod "8a8ecd72-c079-440b-811a-903809ad700e" (UID: "8a8ecd72-c079-440b-811a-903809ad700e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:53.826155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:53.826136 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:53.828182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:53.828150 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-xks-ca" pod "8a8ecd72-c079-440b-811a-903809ad700e" (UID: "8a8ecd72-c079-440b-811a-903809ad700e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:53.828989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:53.828957 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8ecd72-c079-440b-811a-903809ad700e-kube-api-access-mgbxs" pod "8a8ecd72-c079-440b-811a-903809ad700e" (UID: "8a8ecd72-c079-440b-811a-903809ad700e"). InnerVolumeSpecName "kube-api-access-mgbxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:53.928100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:53.928062 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgbxs\" (UniqueName: \"kubernetes.io/projected/8a8ecd72-c079-440b-811a-903809ad700e-kube-api-access-mgbxs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:53.928100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:53.928095 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a8ecd72-c079-440b-811a-903809ad700e-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:08:54.629959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:54.627237 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" event={"ID":"8a8ecd72-c079-440b-811a-903809ad700e","Type":"ContainerDied","Data":"83e6dd0248fbecba17ec8da6128e70091574ff64575dec9c9a646353953d5389"} Oct 30 18:08:54.629959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:54.627309 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83e6dd0248fbecba17ec8da6128e70091574ff64575dec9c9a646353953d5389" Oct 30 18:08:54.629959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:54.627380 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz" Oct 30 18:08:57.282057 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.282008 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm"] Oct 30 18:08:57.306420 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.306389 4883 cpu_manager.go:414] "containerMap: removing container" podUID="8a8ecd72-c079-440b-811a-903809ad700e" containerName="revoke-keys" Oct 30 18:08:57.306668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.306649 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8ecd72-c079-440b-811a-903809ad700e" containerName="revoke-keys" Oct 30 18:08:57.331208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.331165 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8ecd72-c079-440b-811a-903809ad700e" containerName="revoke-keys" Oct 30 18:08:57.426970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.404229 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.474960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.468586 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm"] Oct 30 18:08:57.541018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.539702 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-home\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.541018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.539781 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-tmp-dir\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.541018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.539841 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kchwc\" (UniqueName: \"kubernetes.io/projected/c49b507f-a09f-4f74-8446-1cfc76f2418e-kube-api-access-kchwc\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.541018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.539907 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49b507f-a09f-4f74-8446-1cfc76f2418e-tls-certs\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.541018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.540022 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-model-cache\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.541018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.540061 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-dshm\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.541018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.540328 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3cc458\"/\"distinct-2-w3-3cc458-kserve-self-signed-certs\"" Oct 30 18:08:57.659652 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.647495 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-home\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.757561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.757518 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-tmp-dir\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.757759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.757628 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kchwc\" (UniqueName: \"kubernetes.io/projected/c49b507f-a09f-4f74-8446-1cfc76f2418e-kube-api-access-kchwc\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.757759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.757722 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49b507f-a09f-4f74-8446-1cfc76f2418e-tls-certs\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.757880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.757862 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-model-cache\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.774973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.663556 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-home\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.774973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.769685 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-tmp-dir\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.810376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.757928 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-dshm\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.830958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.826428 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-model-cache\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.862126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.855620 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-dshm\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.943110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.938741 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kchwc\" (UniqueName: \"kubernetes.io/projected/c49b507f-a09f-4f74-8446-1cfc76f2418e-kube-api-access-kchwc\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:57.961621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:57.961535 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49b507f-a09f-4f74-8446-1cfc76f2418e-tls-certs\") pod \"distinct-2-w3-3cc458-kserve-784f5879b5-k68zm\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:58.044985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:58.026393 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:08:58.147972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:58.147136 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:58.404037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:58.392028 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:08:58.404037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:58.392747 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:08:58.404037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:58.393106 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:58.404037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:58.393408 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:08:58.436971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:58.433747 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:08:58.436971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:58.433816 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:58.911108 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:58.911052 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.037034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:59.032348 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.128965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:59.099581 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm"] Oct 30 18:08:59.323789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:59.323747 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.416963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:59.395445 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.416963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:59.395862 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:08:59.416963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:59.396509 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:08:59.416963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:59.396565 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:08:59.416963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:59.409035 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:08:59.416963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:59.409515 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:08:59.493964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:08:59.482704 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:09:00.025958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:00.015033 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:00.152561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:00.151318 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:09:00.382555 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:00.382499 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:00.448823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:00.423871 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:00.448823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:00.441211 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:00.458155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:00.458115 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" event={"ID":"c49b507f-a09f-4f74-8446-1cfc76f2418e","Type":"ContainerStarted","Data":"98a2360bda35a3c272073abbab4d519f3d7c7856732b6b8f00014399e5eaadaa"} Oct 30 18:09:00.611965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:00.611605 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:00.818798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:00.818693 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:09:01.165145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:01.165112 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:09:01.466626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:01.466518 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" event={"ID":"c49b507f-a09f-4f74-8446-1cfc76f2418e","Type":"ContainerStarted","Data":"c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272"} Oct 30 18:09:01.516186 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:01.516119 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podStartSLOduration=4.51609563 podStartE2EDuration="4.51609563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:01.511334932 +0000 UTC m=+1000.468338769" watchObservedRunningTime="2026-10-30 18:09:01.51609563 +0000 UTC m=+1000.473099466" Oct 30 18:09:02.027270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:02.027227 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:09:05.080989 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:09:05.080925 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:09:08.027190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:08.027147 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:09:08.040599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:08.040566 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:09:08.527461 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:08.527430 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:09:09.702231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.702185 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx"] Oct 30 18:09:09.706815 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.706788 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.729380 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.729335 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.729571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.729486 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.729571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.729527 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wncd\" (UniqueName: \"kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-kube-api-access-4wncd\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.729571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.729557 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.729762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.729584 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.729762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.729631 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.729762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.729758 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.729925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.729788 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.729925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.729832 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.737401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.737367 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx"] Oct 30 18:09:09.830791 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.830754 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.830852 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.830888 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wncd\" (UniqueName: \"kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-kube-api-access-4wncd\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.830908 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.830924 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.831012 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.831130 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.831163 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.831202 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.831294 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-credential-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.831322 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-certs\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.831523 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-data\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.831608 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-socket\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.831987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.831963 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istiod-ca-cert\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.833475 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.833447 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-envoy\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.833650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.833629 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-podinfo\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.866845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.866811 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wncd\" (UniqueName: \"kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-kube-api-access-4wncd\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:09.867451 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:09.867426 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-token\") pod \"e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:10.023917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:10.023832 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:10.353031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:10.352998 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx"] Oct 30 18:09:10.353833 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:09:10.353806 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b1ce30_99a1_4d3b_8bb1_f41c9705a9b9.slice/crio-c827f282f8daa7a00b2340a016383cd2511c2bdff2f5dc04ec2a87a2d5c6fbdf WatchSource:0}: Error finding container c827f282f8daa7a00b2340a016383cd2511c2bdff2f5dc04ec2a87a2d5c6fbdf: Status 404 returned error can't find the container with id c827f282f8daa7a00b2340a016383cd2511c2bdff2f5dc04ec2a87a2d5c6fbdf Oct 30 18:09:10.356258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:10.356225 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:09:10.356323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:10.356291 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:09:10.356323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:10.356317 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:09:10.536437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:10.536400 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" event={"ID":"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9","Type":"ContainerStarted","Data":"33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226"} Oct 30 18:09:10.536616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:10.536457 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" event={"ID":"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9","Type":"ContainerStarted","Data":"c827f282f8daa7a00b2340a016383cd2511c2bdff2f5dc04ec2a87a2d5c6fbdf"} Oct 30 18:09:11.024437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:11.024394 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:11.026111 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:11.026081 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:09:11.026244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:11.026132 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:09:12.024520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:12.024482 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:09:12.024984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:12.024541 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:09:13.024674 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:13.024636 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:09:13.025187 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:13.024717 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:09:14.024429 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:14.024391 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:09:14.024652 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:14.024454 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:09:15.024714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:15.024678 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:09:15.025241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:15.024741 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:09:16.024852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:16.024813 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:09:16.025303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:16.024886 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:09:17.025302 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:17.025260 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:09:17.025841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:17.025329 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:09:18.036011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.035977 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:18.036422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.036273 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:18.037407 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.037384 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:09:18.100213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.100146 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" podStartSLOduration=9.100121859 podStartE2EDuration="9.100121859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:10.585185892 +0000 UTC m=+1009.542189730" watchObservedRunningTime="2026-10-30 18:09:18.100121859 +0000 UTC m=+1017.057125697" Oct 30 18:09:18.643720 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.643665 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb"] Oct 30 18:09:18.649014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.648980 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.674233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.674193 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb"] Oct 30 18:09:18.683392 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.683356 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6e8fca\"/\"distinct-w2-6e8fca-kserve-self-signed-certs\"" Oct 30 18:09:18.786614 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.786573 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-dshm\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.786614 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.786615 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-home\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.786879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.786641 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tls-certs\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.786973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.786876 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tmp-dir\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.787037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.786997 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2h2\" (UniqueName: \"kubernetes.io/projected/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-kube-api-access-jh2h2\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.787177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.787149 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-model-cache\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.890069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.889434 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-dshm\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.890069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.889485 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-home\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.890069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.889515 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tls-certs\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.890069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.889595 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tmp-dir\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.890069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.889646 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2h2\" (UniqueName: \"kubernetes.io/projected/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-kube-api-access-jh2h2\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.890069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.889747 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-model-cache\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.891191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.890844 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-model-cache\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.892958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.892276 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tmp-dir\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.892958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.892876 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-home\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.900608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.900488 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-dshm\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.912319 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.912288 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tls-certs\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.928076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.926999 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2h2\" (UniqueName: \"kubernetes.io/projected/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-kube-api-access-jh2h2\") pod \"distinct-w2-6e8fca-kserve-5969cc768d-tn7hb\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:18.969530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:18.969488 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:19.302846 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:19.302815 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb"] Oct 30 18:09:19.303798 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:09:19.303768 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod508b44e7_8aef_4e22_84af_8efaa2b9c5d6.slice/crio-c6689a707a0ef05c975a8847f54d507d6fe6f833ad86a050bdedd7ab5ec3d343 WatchSource:0}: Error finding container c6689a707a0ef05c975a8847f54d507d6fe6f833ad86a050bdedd7ab5ec3d343: Status 404 returned error can't find the container with id c6689a707a0ef05c975a8847f54d507d6fe6f833ad86a050bdedd7ab5ec3d343 Oct 30 18:09:19.620414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:19.620370 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" event={"ID":"508b44e7-8aef-4e22-84af-8efaa2b9c5d6","Type":"ContainerStarted","Data":"0a6d2485d783be3c58f1ea8390c413342934380c4bdf352ca018fbf800fdc2c8"} Oct 30 18:09:19.620414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:19.620426 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" event={"ID":"508b44e7-8aef-4e22-84af-8efaa2b9c5d6","Type":"ContainerStarted","Data":"c6689a707a0ef05c975a8847f54d507d6fe6f833ad86a050bdedd7ab5ec3d343"} Oct 30 18:09:19.664884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:19.664829 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podStartSLOduration=1.664814476 podStartE2EDuration="1.664814476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:19.663381542 +0000 UTC m=+1018.620385378" watchObservedRunningTime="2026-10-30 18:09:19.664814476 +0000 UTC m=+1018.621818311" Oct 30 18:09:19.970489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:19.970403 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:28.969914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:28.969870 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:28.982827 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:28.982797 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:29.703051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:29.703004 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:09:35.080884 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:09:35.080789 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:09:43.995980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:43.995038 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.141544 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:44.140387 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.418118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:44.417516 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.441477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:44.439323 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.442738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:44.442701 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.504961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:44.504200 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:44.517864 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:44.517820 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:45.850251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:45.849218 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:46.081170 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:46.081117 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:46.464969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:46.464037 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:46.628165 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:46.628093 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:46.731156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:46.731041 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:46.787423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:46.731393 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:46.978962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:46.956730 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:47.139727 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:47.139680 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:47.519167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:47.519054 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:09:47.566960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:47.557872 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:09:47.673751 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:47.673701 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:09:47.760898 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:47.760861 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:09:47.868410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:47.868379 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:09:48.035431 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:48.035152 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:09:48.918735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:48.918698 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz"] Oct 30 18:09:48.923220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:48.923194 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:48.935384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:48.935355 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz"] Oct 30 18:09:48.940207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:48.940181 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:48.944099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:48.944069 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:09:48.944265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:48.944188 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:09:48.944331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:48.944271 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:09:48.953114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:48.953067 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:09:48.957555 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:48.957520 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz"] Oct 30 18:09:48.962459 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:48.962428 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz"] Oct 30 18:09:49.037050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.037013 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a6043a3-6qdfz\" (UID: \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:49.037498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.037123 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxkwq\" (UniqueName: \"kubernetes.io/projected/32bf2b1f-efc8-4b29-87df-55d5606d3e46-kube-api-access-mxkwq\") pod \"maas-api-revoke-sub-8a6043a3-6qdfz\" (UID: \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:49.037498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.037181 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a6043a3-6qdfz\" (UID: \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:49.037498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.037238 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g6rx\" (UniqueName: \"kubernetes.io/projected/b7d89d01-b717-4336-9951-620ef9dce005-kube-api-access-4g6rx\") pod \"maas-api-revoke-sub-dee44a3b-x28hz\" (UID: \"b7d89d01-b717-4336-9951-620ef9dce005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:49.037498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.037282 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dee44a3b-x28hz\" (UID: \"b7d89d01-b717-4336-9951-620ef9dce005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:49.037498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.037487 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dee44a3b-x28hz\" (UID: \"b7d89d01-b717-4336-9951-620ef9dce005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:49.138840 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.138796 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a6043a3-6qdfz\" (UID: \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:49.139067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.138874 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkwq\" (UniqueName: \"kubernetes.io/projected/32bf2b1f-efc8-4b29-87df-55d5606d3e46-kube-api-access-mxkwq\") pod \"maas-api-revoke-sub-8a6043a3-6qdfz\" (UID: \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:49.139067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.138908 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a6043a3-6qdfz\" (UID: \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:49.139067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.139025 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g6rx\" (UniqueName: \"kubernetes.io/projected/b7d89d01-b717-4336-9951-620ef9dce005-kube-api-access-4g6rx\") pod \"maas-api-revoke-sub-dee44a3b-x28hz\" (UID: \"b7d89d01-b717-4336-9951-620ef9dce005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:49.139259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.139087 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dee44a3b-x28hz\" (UID: \"b7d89d01-b717-4336-9951-620ef9dce005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:49.139259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.139202 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dee44a3b-x28hz\" (UID: \"b7d89d01-b717-4336-9951-620ef9dce005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:49.139827 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.139797 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a6043a3-6qdfz\" (UID: \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:49.139980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.139838 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dee44a3b-x28hz\" (UID: \"b7d89d01-b717-4336-9951-620ef9dce005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:49.141524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.141502 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a6043a3-6qdfz\" (UID: \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:49.141615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.141572 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dee44a3b-x28hz\" (UID: \"b7d89d01-b717-4336-9951-620ef9dce005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:49.163913 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.163874 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g6rx\" (UniqueName: \"kubernetes.io/projected/b7d89d01-b717-4336-9951-620ef9dce005-kube-api-access-4g6rx\") pod \"maas-api-revoke-sub-dee44a3b-x28hz\" (UID: \"b7d89d01-b717-4336-9951-620ef9dce005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:49.164096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.163872 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkwq\" (UniqueName: \"kubernetes.io/projected/32bf2b1f-efc8-4b29-87df-55d5606d3e46-kube-api-access-mxkwq\") pod \"maas-api-revoke-sub-8a6043a3-6qdfz\" (UID: \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:49.240045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.239964 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:49.257465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.257431 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:49.567445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.567411 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz"] Oct 30 18:09:49.568793 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:09:49.568758 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d89d01_b717_4336_9951_620ef9dce005.slice/crio-90d1538b266e1a1b2aa3bf687fc097ec3d21237475acfaede1ea53059f5665c1 WatchSource:0}: Error finding container 90d1538b266e1a1b2aa3bf687fc097ec3d21237475acfaede1ea53059f5665c1: Status 404 returned error can't find the container with id 90d1538b266e1a1b2aa3bf687fc097ec3d21237475acfaede1ea53059f5665c1 Oct 30 18:09:49.586068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:49.586031 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz"] Oct 30 18:09:49.588337 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:09:49.588301 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32bf2b1f_efc8_4b29_87df_55d5606d3e46.slice/crio-bdc055943f14c0956b830feb6910a045a3e9c2d5d6030b06cf31f68636f6d9ae WatchSource:0}: Error finding container bdc055943f14c0956b830feb6910a045a3e9c2d5d6030b06cf31f68636f6d9ae: Status 404 returned error can't find the container with id bdc055943f14c0956b830feb6910a045a3e9c2d5d6030b06cf31f68636f6d9ae Oct 30 18:09:50.459528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:50.459490 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" event={"ID":"b7d89d01-b717-4336-9951-620ef9dce005","Type":"ContainerStarted","Data":"bffed7ea29f11ac814ec1d39cb1b9b569eaf4d3015f527b04a5ef6832eb74a00"} Oct 30 18:09:50.460028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:50.459551 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" event={"ID":"b7d89d01-b717-4336-9951-620ef9dce005","Type":"ContainerStarted","Data":"90d1538b266e1a1b2aa3bf687fc097ec3d21237475acfaede1ea53059f5665c1"} Oct 30 18:09:50.461243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:50.461211 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" event={"ID":"32bf2b1f-efc8-4b29-87df-55d5606d3e46","Type":"ContainerStarted","Data":"06a89084991f43e24bdafd29d19d2944a1669d01874f643b94f48d6fe019c6df"} Oct 30 18:09:50.461366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:50.461269 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" event={"ID":"32bf2b1f-efc8-4b29-87df-55d5606d3e46","Type":"ContainerStarted","Data":"bdc055943f14c0956b830feb6910a045a3e9c2d5d6030b06cf31f68636f6d9ae"} Oct 30 18:09:50.505033 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:50.504973 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" podStartSLOduration=2.5049583 podStartE2EDuration="2.5049583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:50.501904739 +0000 UTC m=+1049.458908577" watchObservedRunningTime="2026-10-30 18:09:50.5049583 +0000 UTC m=+1049.461962133" Oct 30 18:09:50.547088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:50.547032 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" podStartSLOduration=2.547015837 podStartE2EDuration="2.547015837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:50.544836803 +0000 UTC m=+1049.501840636" watchObservedRunningTime="2026-10-30 18:09:50.547015837 +0000 UTC m=+1049.504019672" Oct 30 18:09:51.470148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:51.470112 4883 generic.go:356] "Generic (PLEG): container finished" podID="b7d89d01-b717-4336-9951-620ef9dce005" containerID="bffed7ea29f11ac814ec1d39cb1b9b569eaf4d3015f527b04a5ef6832eb74a00" exitCode=0 Oct 30 18:09:51.470615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:51.470193 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" event={"ID":"b7d89d01-b717-4336-9951-620ef9dce005","Type":"ContainerDied","Data":"bffed7ea29f11ac814ec1d39cb1b9b569eaf4d3015f527b04a5ef6832eb74a00"} Oct 30 18:09:51.471863 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:51.471838 4883 generic.go:356] "Generic (PLEG): container finished" podID="32bf2b1f-efc8-4b29-87df-55d5606d3e46" containerID="06a89084991f43e24bdafd29d19d2944a1669d01874f643b94f48d6fe019c6df" exitCode=0 Oct 30 18:09:51.471981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:51.471927 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" event={"ID":"32bf2b1f-efc8-4b29-87df-55d5606d3e46","Type":"ContainerDied","Data":"06a89084991f43e24bdafd29d19d2944a1669d01874f643b94f48d6fe019c6df"} Oct 30 18:09:52.662279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.662251 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:52.724439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.724406 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:52.796916 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.796785 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-xks-ca\") pod \"b7d89d01-b717-4336-9951-620ef9dce005\" (UID: \"b7d89d01-b717-4336-9951-620ef9dce005\") " Oct 30 18:09:52.797189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.796922 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-openshift-ca\") pod \"b7d89d01-b717-4336-9951-620ef9dce005\" (UID: \"b7d89d01-b717-4336-9951-620ef9dce005\") " Oct 30 18:09:52.797189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.797028 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32bf2b1f-efc8-4b29-87df-55d5606d3e46-kube-api-access-mxkwq\" (UniqueName: \"kubernetes.io/projected/32bf2b1f-efc8-4b29-87df-55d5606d3e46-kube-api-access-mxkwq\") pod \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\" (UID: \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\") " Oct 30 18:09:52.797189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.797084 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-openshift-ca\") pod \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\" (UID: \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\") " Oct 30 18:09:52.798001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.797946 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-openshift-ca" pod "b7d89d01-b717-4336-9951-620ef9dce005" (UID: "b7d89d01-b717-4336-9951-620ef9dce005"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:52.798109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.798016 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-openshift-ca" pod "32bf2b1f-efc8-4b29-87df-55d5606d3e46" (UID: "32bf2b1f-efc8-4b29-87df-55d5606d3e46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:52.798109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.798050 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-xks-ca\") pod \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\" (UID: \"32bf2b1f-efc8-4b29-87df-55d5606d3e46\") " Oct 30 18:09:52.798218 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.798192 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d89d01-b717-4336-9951-620ef9dce005-kube-api-access-4g6rx\" (UniqueName: \"kubernetes.io/projected/b7d89d01-b717-4336-9951-620ef9dce005-kube-api-access-4g6rx\") pod \"b7d89d01-b717-4336-9951-620ef9dce005\" (UID: \"b7d89d01-b717-4336-9951-620ef9dce005\") " Oct 30 18:09:52.799083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.799040 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:09:52.799083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.799068 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:09:52.800484 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.800453 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-xks-ca" pod "b7d89d01-b717-4336-9951-620ef9dce005" (UID: "b7d89d01-b717-4336-9951-620ef9dce005"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:52.800673 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.800645 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32bf2b1f-efc8-4b29-87df-55d5606d3e46-kube-api-access-mxkwq" pod "32bf2b1f-efc8-4b29-87df-55d5606d3e46" (UID: "32bf2b1f-efc8-4b29-87df-55d5606d3e46"). InnerVolumeSpecName "kube-api-access-mxkwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:52.801357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.801327 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-xks-ca" pod "32bf2b1f-efc8-4b29-87df-55d5606d3e46" (UID: "32bf2b1f-efc8-4b29-87df-55d5606d3e46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:52.801479 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.801454 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d89d01-b717-4336-9951-620ef9dce005-kube-api-access-4g6rx" pod "b7d89d01-b717-4336-9951-620ef9dce005" (UID: "b7d89d01-b717-4336-9951-620ef9dce005"). InnerVolumeSpecName "kube-api-access-4g6rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:52.900039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.899994 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxkwq\" (UniqueName: \"kubernetes.io/projected/32bf2b1f-efc8-4b29-87df-55d5606d3e46-kube-api-access-mxkwq\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:09:52.900039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.900030 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32bf2b1f-efc8-4b29-87df-55d5606d3e46-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:09:52.900265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.900052 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g6rx\" (UniqueName: \"kubernetes.io/projected/b7d89d01-b717-4336-9951-620ef9dce005-kube-api-access-4g6rx\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:09:52.900265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:52.900065 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d89d01-b717-4336-9951-620ef9dce005-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:09:53.491056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:53.490959 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" Oct 30 18:09:53.491056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:53.491001 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz" event={"ID":"b7d89d01-b717-4336-9951-620ef9dce005","Type":"ContainerDied","Data":"90d1538b266e1a1b2aa3bf687fc097ec3d21237475acfaede1ea53059f5665c1"} Oct 30 18:09:53.491276 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:53.491085 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90d1538b266e1a1b2aa3bf687fc097ec3d21237475acfaede1ea53059f5665c1" Oct 30 18:09:53.492826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:53.492742 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" Oct 30 18:09:53.492826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:53.492751 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz" event={"ID":"32bf2b1f-efc8-4b29-87df-55d5606d3e46","Type":"ContainerDied","Data":"bdc055943f14c0956b830feb6910a045a3e9c2d5d6030b06cf31f68636f6d9ae"} Oct 30 18:09:53.492826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:09:53.492807 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdc055943f14c0956b830feb6910a045a3e9c2d5d6030b06cf31f68636f6d9ae" Oct 30 18:10:02.410993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.410949 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d"] Oct 30 18:10:02.411504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.411489 4883 cpu_manager.go:414] "containerMap: removing container" podUID="b7d89d01-b717-4336-9951-620ef9dce005" containerName="revoke-keys" Oct 30 18:10:02.411549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.411506 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d89d01-b717-4336-9951-620ef9dce005" containerName="revoke-keys" Oct 30 18:10:02.411549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.411543 4883 cpu_manager.go:414] "containerMap: removing container" podUID="32bf2b1f-efc8-4b29-87df-55d5606d3e46" containerName="revoke-keys" Oct 30 18:10:02.411615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.411549 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32bf2b1f-efc8-4b29-87df-55d5606d3e46" containerName="revoke-keys" Oct 30 18:10:02.411647 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.411637 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d89d01-b717-4336-9951-620ef9dce005" containerName="revoke-keys" Oct 30 18:10:02.411681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.411651 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="32bf2b1f-efc8-4b29-87df-55d5606d3e46" containerName="revoke-keys" Oct 30 18:10:02.416531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.416499 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:02.437699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.437667 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d"] Oct 30 18:10:02.446310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.446274 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:10:02.446473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.446318 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:10:02.446473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.446323 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:10:02.447465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.447445 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:10:02.530426 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.530391 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7075388e-dq94d\" (UID: \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:02.530426 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.530428 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdpxd\" (UniqueName: \"kubernetes.io/projected/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-kube-api-access-mdpxd\") pod \"maas-api-revoke-sub-7075388e-dq94d\" (UID: \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:02.530648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.530550 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7075388e-dq94d\" (UID: \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:02.631691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.631655 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7075388e-dq94d\" (UID: \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:02.631691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.631693 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdpxd\" (UniqueName: \"kubernetes.io/projected/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-kube-api-access-mdpxd\") pod \"maas-api-revoke-sub-7075388e-dq94d\" (UID: \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:02.631902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.631771 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7075388e-dq94d\" (UID: \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:02.632408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.632387 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7075388e-dq94d\" (UID: \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:02.634201 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.634173 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7075388e-dq94d\" (UID: \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:02.656026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.655997 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdpxd\" (UniqueName: \"kubernetes.io/projected/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-kube-api-access-mdpxd\") pod \"maas-api-revoke-sub-7075388e-dq94d\" (UID: \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:02.730763 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:02.730660 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:03.042716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:03.042629 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d"] Oct 30 18:10:03.046780 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:10:03.046747 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb0e632_4a21_4909_9d4c_8eb6d0d8135c.slice/crio-75bb3568fe15ebc541616b6f858fa32ee2683ffd3856877f56888dca36115116 WatchSource:0}: Error finding container 75bb3568fe15ebc541616b6f858fa32ee2683ffd3856877f56888dca36115116: Status 404 returned error can't find the container with id 75bb3568fe15ebc541616b6f858fa32ee2683ffd3856877f56888dca36115116 Oct 30 18:10:03.580658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:03.580610 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" event={"ID":"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c","Type":"ContainerStarted","Data":"9fbaf758eda3c4b614e5d74386f246ae8f379c401a880b4154866897fb17a5f9"} Oct 30 18:10:03.581122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:03.580673 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" event={"ID":"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c","Type":"ContainerStarted","Data":"75bb3568fe15ebc541616b6f858fa32ee2683ffd3856877f56888dca36115116"} Oct 30 18:10:03.623375 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:03.623307 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" podStartSLOduration=1.623291315 podStartE2EDuration="1.623291315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:03.622763814 +0000 UTC m=+1062.579767651" watchObservedRunningTime="2026-10-30 18:10:03.623291315 +0000 UTC m=+1062.580295151" Oct 30 18:10:04.591703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:04.591667 4883 generic.go:356] "Generic (PLEG): container finished" podID="2fb0e632-4a21-4909-9d4c-8eb6d0d8135c" containerID="9fbaf758eda3c4b614e5d74386f246ae8f379c401a880b4154866897fb17a5f9" exitCode=0 Oct 30 18:10:04.592164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:04.591756 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" event={"ID":"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c","Type":"ContainerDied","Data":"9fbaf758eda3c4b614e5d74386f246ae8f379c401a880b4154866897fb17a5f9"} Oct 30 18:10:05.088554 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:10:05.088511 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:10:05.766649 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:05.766619 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:05.909265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:05.909176 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-openshift-ca\") pod \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\" (UID: \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\") " Oct 30 18:10:05.909428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:05.909406 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-kube-api-access-mdpxd\" (UniqueName: \"kubernetes.io/projected/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-kube-api-access-mdpxd\") pod \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\" (UID: \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\") " Oct 30 18:10:05.909532 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:05.909514 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-xks-ca\") pod \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\" (UID: \"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c\") " Oct 30 18:10:05.909635 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:05.909614 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-openshift-ca" pod "2fb0e632-4a21-4909-9d4c-8eb6d0d8135c" (UID: "2fb0e632-4a21-4909-9d4c-8eb6d0d8135c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:05.909965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:05.909950 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:05.911661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:05.911634 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-xks-ca" pod "2fb0e632-4a21-4909-9d4c-8eb6d0d8135c" (UID: "2fb0e632-4a21-4909-9d4c-8eb6d0d8135c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:05.911747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:05.911671 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-kube-api-access-mdpxd" pod "2fb0e632-4a21-4909-9d4c-8eb6d0d8135c" (UID: "2fb0e632-4a21-4909-9d4c-8eb6d0d8135c"). InnerVolumeSpecName "kube-api-access-mdpxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:06.011669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:06.011625 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:06.011669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:06.011664 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdpxd\" (UniqueName: \"kubernetes.io/projected/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c-kube-api-access-mdpxd\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:06.610454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:06.610423 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" Oct 30 18:10:06.610629 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:06.610459 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d" event={"ID":"2fb0e632-4a21-4909-9d4c-8eb6d0d8135c","Type":"ContainerDied","Data":"75bb3568fe15ebc541616b6f858fa32ee2683ffd3856877f56888dca36115116"} Oct 30 18:10:06.610629 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:06.610510 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75bb3568fe15ebc541616b6f858fa32ee2683ffd3856877f56888dca36115116" Oct 30 18:10:10.428384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.428345 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw"] Oct 30 18:10:10.429023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.429007 4883 cpu_manager.go:414] "containerMap: removing container" podUID="2fb0e632-4a21-4909-9d4c-8eb6d0d8135c" containerName="revoke-keys" Oct 30 18:10:10.429118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.429025 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb0e632-4a21-4909-9d4c-8eb6d0d8135c" containerName="revoke-keys" Oct 30 18:10:10.429166 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.429121 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb0e632-4a21-4909-9d4c-8eb6d0d8135c" containerName="revoke-keys" Oct 30 18:10:10.432683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.432655 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:10.469987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.462786 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:10:10.469987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.462836 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:10:10.469987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.462911 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:10:10.469987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.468031 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:10:10.469987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.469743 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw"] Oct 30 18:10:10.575653 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.575609 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6bcfc8-jzlmw\" (UID: \"19372b31-927c-4e58-b860-14b49d960fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:10.575839 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.575682 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6bcfc8-jzlmw\" (UID: \"19372b31-927c-4e58-b860-14b49d960fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:10.575839 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.575790 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ctd6\" (UniqueName: \"kubernetes.io/projected/19372b31-927c-4e58-b860-14b49d960fe7-kube-api-access-6ctd6\") pod \"maas-api-revoke-sub-9d6bcfc8-jzlmw\" (UID: \"19372b31-927c-4e58-b860-14b49d960fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:10.677258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.677218 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6bcfc8-jzlmw\" (UID: \"19372b31-927c-4e58-b860-14b49d960fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:10.677412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.677361 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ctd6\" (UniqueName: \"kubernetes.io/projected/19372b31-927c-4e58-b860-14b49d960fe7-kube-api-access-6ctd6\") pod \"maas-api-revoke-sub-9d6bcfc8-jzlmw\" (UID: \"19372b31-927c-4e58-b860-14b49d960fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:10.677613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.677595 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6bcfc8-jzlmw\" (UID: \"19372b31-927c-4e58-b860-14b49d960fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:10.678242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.678222 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d6bcfc8-jzlmw\" (UID: \"19372b31-927c-4e58-b860-14b49d960fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:10.679767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.679723 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d6bcfc8-jzlmw\" (UID: \"19372b31-927c-4e58-b860-14b49d960fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:10.702223 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.702194 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ctd6\" (UniqueName: \"kubernetes.io/projected/19372b31-927c-4e58-b860-14b49d960fe7-kube-api-access-6ctd6\") pod \"maas-api-revoke-sub-9d6bcfc8-jzlmw\" (UID: \"19372b31-927c-4e58-b860-14b49d960fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:10.748461 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:10.748421 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:11.058779 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:11.058742 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw"] Oct 30 18:10:11.060304 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:10:11.060272 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19372b31_927c_4e58_b860_14b49d960fe7.slice/crio-fd2bf39b33cfd7e1dda468fcf377f2d6ff122cff09eef2f3fab7bb0f6fc46c5d WatchSource:0}: Error finding container fd2bf39b33cfd7e1dda468fcf377f2d6ff122cff09eef2f3fab7bb0f6fc46c5d: Status 404 returned error can't find the container with id fd2bf39b33cfd7e1dda468fcf377f2d6ff122cff09eef2f3fab7bb0f6fc46c5d Oct 30 18:10:11.647766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:11.647666 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" event={"ID":"19372b31-927c-4e58-b860-14b49d960fe7","Type":"ContainerStarted","Data":"0af28b10e433fda33826038fd58d8bc9ef60a2aa79b4c6bf88c073f7a3778540"} Oct 30 18:10:11.647766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:11.647743 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" event={"ID":"19372b31-927c-4e58-b860-14b49d960fe7","Type":"ContainerStarted","Data":"fd2bf39b33cfd7e1dda468fcf377f2d6ff122cff09eef2f3fab7bb0f6fc46c5d"} Oct 30 18:10:11.694167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:11.693832 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" podStartSLOduration=1.693811207 podStartE2EDuration="1.693811207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:11.690411585 +0000 UTC m=+1070.647415422" watchObservedRunningTime="2026-10-30 18:10:11.693811207 +0000 UTC m=+1070.650815044" Oct 30 18:10:12.656276 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:12.656244 4883 generic.go:356] "Generic (PLEG): container finished" podID="19372b31-927c-4e58-b860-14b49d960fe7" containerID="0af28b10e433fda33826038fd58d8bc9ef60a2aa79b4c6bf88c073f7a3778540" exitCode=0 Oct 30 18:10:12.656692 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:12.656327 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" event={"ID":"19372b31-927c-4e58-b860-14b49d960fe7","Type":"ContainerDied","Data":"0af28b10e433fda33826038fd58d8bc9ef60a2aa79b4c6bf88c073f7a3778540"} Oct 30 18:10:12.833286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:12.833239 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww"] Oct 30 18:10:12.837266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:12.837240 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:12.858521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:12.858488 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww"] Oct 30 18:10:12.909006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:12.908874 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78ch2\" (UniqueName: \"kubernetes.io/projected/0816c7e7-f68b-49f6-bd92-8632a42f9865-kube-api-access-78ch2\") pod \"maas-api-revoke-sub-f7eb175f-df7ww\" (UID: \"0816c7e7-f68b-49f6-bd92-8632a42f9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:12.909223 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:12.909200 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7eb175f-df7ww\" (UID: \"0816c7e7-f68b-49f6-bd92-8632a42f9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:12.909290 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:12.909282 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7eb175f-df7ww\" (UID: \"0816c7e7-f68b-49f6-bd92-8632a42f9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:13.010153 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.010120 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7eb175f-df7ww\" (UID: \"0816c7e7-f68b-49f6-bd92-8632a42f9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:13.010316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.010185 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7eb175f-df7ww\" (UID: \"0816c7e7-f68b-49f6-bd92-8632a42f9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:13.010316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.010256 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78ch2\" (UniqueName: \"kubernetes.io/projected/0816c7e7-f68b-49f6-bd92-8632a42f9865-kube-api-access-78ch2\") pod \"maas-api-revoke-sub-f7eb175f-df7ww\" (UID: \"0816c7e7-f68b-49f6-bd92-8632a42f9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:13.011069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.011040 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7eb175f-df7ww\" (UID: \"0816c7e7-f68b-49f6-bd92-8632a42f9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:13.012648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.012617 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7eb175f-df7ww\" (UID: \"0816c7e7-f68b-49f6-bd92-8632a42f9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:13.038247 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.038209 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78ch2\" (UniqueName: \"kubernetes.io/projected/0816c7e7-f68b-49f6-bd92-8632a42f9865-kube-api-access-78ch2\") pod \"maas-api-revoke-sub-f7eb175f-df7ww\" (UID: \"0816c7e7-f68b-49f6-bd92-8632a42f9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:13.154627 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.154592 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:13.165140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.165072 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb"] Oct 30 18:10:13.169756 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.169479 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:13.191828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.191794 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb"] Oct 30 18:10:13.212994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.211917 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b8125f9-swpxb\" (UID: \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:13.212994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.212203 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b8125f9-swpxb\" (UID: \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:13.212994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.212297 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ncgh\" (UniqueName: \"kubernetes.io/projected/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-kube-api-access-8ncgh\") pod \"maas-api-revoke-sub-2b8125f9-swpxb\" (UID: \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:13.313925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.313885 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b8125f9-swpxb\" (UID: \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:13.314119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.314028 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b8125f9-swpxb\" (UID: \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:13.314119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.314051 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ncgh\" (UniqueName: \"kubernetes.io/projected/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-kube-api-access-8ncgh\") pod \"maas-api-revoke-sub-2b8125f9-swpxb\" (UID: \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:13.314737 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.314710 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b8125f9-swpxb\" (UID: \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:13.316555 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.316512 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b8125f9-swpxb\" (UID: \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:13.339354 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.339324 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ncgh\" (UniqueName: \"kubernetes.io/projected/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-kube-api-access-8ncgh\") pod \"maas-api-revoke-sub-2b8125f9-swpxb\" (UID: \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:13.477447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.477408 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww"] Oct 30 18:10:13.478377 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:10:13.478349 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0816c7e7_f68b_49f6_bd92_8632a42f9865.slice/crio-565df8a3f4982b19b5aed009288cdd1d68abe1b3504bf6fd27c66fcc7576824b WatchSource:0}: Error finding container 565df8a3f4982b19b5aed009288cdd1d68abe1b3504bf6fd27c66fcc7576824b: Status 404 returned error can't find the container with id 565df8a3f4982b19b5aed009288cdd1d68abe1b3504bf6fd27c66fcc7576824b Oct 30 18:10:13.510534 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.510485 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:13.665586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.665544 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" event={"ID":"0816c7e7-f68b-49f6-bd92-8632a42f9865","Type":"ContainerStarted","Data":"565df8a3f4982b19b5aed009288cdd1d68abe1b3504bf6fd27c66fcc7576824b"} Oct 30 18:10:13.808806 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.808775 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:13.820928 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.820898 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-openshift-ca\") pod \"19372b31-927c-4e58-b860-14b49d960fe7\" (UID: \"19372b31-927c-4e58-b860-14b49d960fe7\") " Oct 30 18:10:13.821100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.820968 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-xks-ca\") pod \"19372b31-927c-4e58-b860-14b49d960fe7\" (UID: \"19372b31-927c-4e58-b860-14b49d960fe7\") " Oct 30 18:10:13.821139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.821120 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19372b31-927c-4e58-b860-14b49d960fe7-kube-api-access-6ctd6\" (UniqueName: \"kubernetes.io/projected/19372b31-927c-4e58-b860-14b49d960fe7-kube-api-access-6ctd6\") pod \"19372b31-927c-4e58-b860-14b49d960fe7\" (UID: \"19372b31-927c-4e58-b860-14b49d960fe7\") " Oct 30 18:10:13.821898 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.821871 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-openshift-ca" pod "19372b31-927c-4e58-b860-14b49d960fe7" (UID: "19372b31-927c-4e58-b860-14b49d960fe7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:13.824204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.824061 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-xks-ca" pod "19372b31-927c-4e58-b860-14b49d960fe7" (UID: "19372b31-927c-4e58-b860-14b49d960fe7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:13.824204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.824195 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19372b31-927c-4e58-b860-14b49d960fe7-kube-api-access-6ctd6" pod "19372b31-927c-4e58-b860-14b49d960fe7" (UID: "19372b31-927c-4e58-b860-14b49d960fe7"). InnerVolumeSpecName "kube-api-access-6ctd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:13.824327 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.824224 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb"] Oct 30 18:10:13.827619 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:10:13.827593 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbaf21a_9dc7_42da_abd2_7c6ed668963c.slice/crio-a6e4aab65d4457b987b4d9dcd18ba678d5d6c9765a1890b01616afcd44266c5e WatchSource:0}: Error finding container a6e4aab65d4457b987b4d9dcd18ba678d5d6c9765a1890b01616afcd44266c5e: Status 404 returned error can't find the container with id a6e4aab65d4457b987b4d9dcd18ba678d5d6c9765a1890b01616afcd44266c5e Oct 30 18:10:13.923164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.923117 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ctd6\" (UniqueName: \"kubernetes.io/projected/19372b31-927c-4e58-b860-14b49d960fe7-kube-api-access-6ctd6\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:13.923164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.923167 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:13.923378 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:13.923179 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19372b31-927c-4e58-b860-14b49d960fe7-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:14.674242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:14.674209 4883 generic.go:356] "Generic (PLEG): container finished" podID="0816c7e7-f68b-49f6-bd92-8632a42f9865" containerID="a6672df9246fc50af99255782069cbe0c0e478f68ba96460e4fefcc4f344dd63" exitCode=0 Oct 30 18:10:14.674690 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:14.674289 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" event={"ID":"0816c7e7-f68b-49f6-bd92-8632a42f9865","Type":"ContainerDied","Data":"a6672df9246fc50af99255782069cbe0c0e478f68ba96460e4fefcc4f344dd63"} Oct 30 18:10:14.675995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:14.675964 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" event={"ID":"fbbaf21a-9dc7-42da-abd2-7c6ed668963c","Type":"ContainerStarted","Data":"bc0e1c32aa657bfdd254f2568798730f014b81445c9b95717336b53198eb2312"} Oct 30 18:10:14.676126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:14.676022 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" event={"ID":"fbbaf21a-9dc7-42da-abd2-7c6ed668963c","Type":"ContainerStarted","Data":"a6e4aab65d4457b987b4d9dcd18ba678d5d6c9765a1890b01616afcd44266c5e"} Oct 30 18:10:14.677422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:14.677392 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" Oct 30 18:10:14.677525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:14.677436 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw" event={"ID":"19372b31-927c-4e58-b860-14b49d960fe7","Type":"ContainerDied","Data":"fd2bf39b33cfd7e1dda468fcf377f2d6ff122cff09eef2f3fab7bb0f6fc46c5d"} Oct 30 18:10:14.677525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:14.677482 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd2bf39b33cfd7e1dda468fcf377f2d6ff122cff09eef2f3fab7bb0f6fc46c5d" Oct 30 18:10:14.776866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:14.776804 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" podStartSLOduration=1.776787911 podStartE2EDuration="1.776787911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:14.774433235 +0000 UTC m=+1073.731437071" watchObservedRunningTime="2026-10-30 18:10:14.776787911 +0000 UTC m=+1073.733791748" Oct 30 18:10:15.685818 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:15.685787 4883 generic.go:356] "Generic (PLEG): container finished" podID="fbbaf21a-9dc7-42da-abd2-7c6ed668963c" containerID="bc0e1c32aa657bfdd254f2568798730f014b81445c9b95717336b53198eb2312" exitCode=0 Oct 30 18:10:15.686386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:15.685881 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" event={"ID":"fbbaf21a-9dc7-42da-abd2-7c6ed668963c","Type":"ContainerDied","Data":"bc0e1c32aa657bfdd254f2568798730f014b81445c9b95717336b53198eb2312"} Oct 30 18:10:15.855028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:15.854999 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:15.949750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:15.949646 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-xks-ca\") pod \"0816c7e7-f68b-49f6-bd92-8632a42f9865\" (UID: \"0816c7e7-f68b-49f6-bd92-8632a42f9865\") " Oct 30 18:10:15.949968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:15.949799 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-openshift-ca\") pod \"0816c7e7-f68b-49f6-bd92-8632a42f9865\" (UID: \"0816c7e7-f68b-49f6-bd92-8632a42f9865\") " Oct 30 18:10:15.949968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:15.949910 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0816c7e7-f68b-49f6-bd92-8632a42f9865-kube-api-access-78ch2\" (UniqueName: \"kubernetes.io/projected/0816c7e7-f68b-49f6-bd92-8632a42f9865-kube-api-access-78ch2\") pod \"0816c7e7-f68b-49f6-bd92-8632a42f9865\" (UID: \"0816c7e7-f68b-49f6-bd92-8632a42f9865\") " Oct 30 18:10:15.950193 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:15.950168 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-openshift-ca" pod "0816c7e7-f68b-49f6-bd92-8632a42f9865" (UID: "0816c7e7-f68b-49f6-bd92-8632a42f9865"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:15.950463 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:15.950439 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:15.951855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:15.951829 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-xks-ca" pod "0816c7e7-f68b-49f6-bd92-8632a42f9865" (UID: "0816c7e7-f68b-49f6-bd92-8632a42f9865"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:15.951983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:15.951871 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0816c7e7-f68b-49f6-bd92-8632a42f9865-kube-api-access-78ch2" pod "0816c7e7-f68b-49f6-bd92-8632a42f9865" (UID: "0816c7e7-f68b-49f6-bd92-8632a42f9865"). InnerVolumeSpecName "kube-api-access-78ch2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:16.051997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.051957 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78ch2\" (UniqueName: \"kubernetes.io/projected/0816c7e7-f68b-49f6-bd92-8632a42f9865-kube-api-access-78ch2\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:16.051997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.051991 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0816c7e7-f68b-49f6-bd92-8632a42f9865-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:16.694115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.694078 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" Oct 30 18:10:16.694559 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.694111 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww" event={"ID":"0816c7e7-f68b-49f6-bd92-8632a42f9865","Type":"ContainerDied","Data":"565df8a3f4982b19b5aed009288cdd1d68abe1b3504bf6fd27c66fcc7576824b"} Oct 30 18:10:16.694559 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.694203 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="565df8a3f4982b19b5aed009288cdd1d68abe1b3504bf6fd27c66fcc7576824b" Oct 30 18:10:16.927881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.927853 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:16.967230 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.967125 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-xks-ca\") pod \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\" (UID: \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\") " Oct 30 18:10:16.967230 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.967215 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-openshift-ca\") pod \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\" (UID: \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\") " Oct 30 18:10:16.967475 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.967322 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-kube-api-access-8ncgh\" (UniqueName: \"kubernetes.io/projected/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-kube-api-access-8ncgh\") pod \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\" (UID: \"fbbaf21a-9dc7-42da-abd2-7c6ed668963c\") " Oct 30 18:10:16.967581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.967556 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-openshift-ca" pod "fbbaf21a-9dc7-42da-abd2-7c6ed668963c" (UID: "fbbaf21a-9dc7-42da-abd2-7c6ed668963c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:16.967823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.967807 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:16.969433 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.969402 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-kube-api-access-8ncgh" pod "fbbaf21a-9dc7-42da-abd2-7c6ed668963c" (UID: "fbbaf21a-9dc7-42da-abd2-7c6ed668963c"). InnerVolumeSpecName "kube-api-access-8ncgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:16.969433 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:16.969408 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-xks-ca" pod "fbbaf21a-9dc7-42da-abd2-7c6ed668963c" (UID: "fbbaf21a-9dc7-42da-abd2-7c6ed668963c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:17.069126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:17.069084 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:17.069126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:17.069117 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ncgh\" (UniqueName: \"kubernetes.io/projected/fbbaf21a-9dc7-42da-abd2-7c6ed668963c-kube-api-access-8ncgh\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:17.702588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:17.702555 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" Oct 30 18:10:17.702588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:17.702570 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb" event={"ID":"fbbaf21a-9dc7-42da-abd2-7c6ed668963c","Type":"ContainerDied","Data":"a6e4aab65d4457b987b4d9dcd18ba678d5d6c9765a1890b01616afcd44266c5e"} Oct 30 18:10:17.703051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:17.702622 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6e4aab65d4457b987b4d9dcd18ba678d5d6c9765a1890b01616afcd44266c5e" Oct 30 18:10:21.356336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:21.356303 4883 scope.go:122] "RemoveContainer" containerID="c6f9bb5cc6809d726869a310b1c2192a79d5a8daaa2eb1e78804c4d0320085a9" Oct 30 18:10:21.366453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:21.366432 4883 scope.go:122] "RemoveContainer" containerID="3ff88201be32a0da35d57cfc9906ff8f1a602608803727c111ec6f6518addf29" Oct 30 18:10:21.376231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:21.376207 4883 scope.go:122] "RemoveContainer" containerID="658a997edab539c7ea053bdc4fd32a41be4f7c386f9ecabee884b0d47cd0613f" Oct 30 18:10:21.387473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:21.387447 4883 scope.go:122] "RemoveContainer" containerID="e956e8e3737776215a6d33ac9029b6cc25b91f968275769bb6ee184dcaa0b0ef" Oct 30 18:10:22.807055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.807020 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t"] Oct 30 18:10:22.807827 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.807810 4883 cpu_manager.go:414] "containerMap: removing container" podUID="19372b31-927c-4e58-b860-14b49d960fe7" containerName="revoke-keys" Oct 30 18:10:22.807871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.807830 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19372b31-927c-4e58-b860-14b49d960fe7" containerName="revoke-keys" Oct 30 18:10:22.807871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.807846 4883 cpu_manager.go:414] "containerMap: removing container" podUID="0816c7e7-f68b-49f6-bd92-8632a42f9865" containerName="revoke-keys" Oct 30 18:10:22.807871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.807854 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0816c7e7-f68b-49f6-bd92-8632a42f9865" containerName="revoke-keys" Oct 30 18:10:22.807871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.807865 4883 cpu_manager.go:414] "containerMap: removing container" podUID="fbbaf21a-9dc7-42da-abd2-7c6ed668963c" containerName="revoke-keys" Oct 30 18:10:22.808034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.807874 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbaf21a-9dc7-42da-abd2-7c6ed668963c" containerName="revoke-keys" Oct 30 18:10:22.808080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.808067 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="19372b31-927c-4e58-b860-14b49d960fe7" containerName="revoke-keys" Oct 30 18:10:22.808118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.808083 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="0816c7e7-f68b-49f6-bd92-8632a42f9865" containerName="revoke-keys" Oct 30 18:10:22.808118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.808091 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbaf21a-9dc7-42da-abd2-7c6ed668963c" containerName="revoke-keys" Oct 30 18:10:22.813893 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.813868 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:22.832609 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.832572 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t"] Oct 30 18:10:22.842012 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.841913 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:10:22.842189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.842051 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:10:22.842189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.842108 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:10:22.843609 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.843574 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:10:22.965510 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.965463 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555cb312-wxk9t\" (UID: \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:22.965721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.965522 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ng7t\" (UniqueName: \"kubernetes.io/projected/5f23f75f-fd4c-4592-84d5-ef0a20be658e-kube-api-access-4ng7t\") pod \"maas-api-revoke-sub-555cb312-wxk9t\" (UID: \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:22.965721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:22.965617 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555cb312-wxk9t\" (UID: \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:23.066468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.066386 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555cb312-wxk9t\" (UID: \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:23.066468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.066447 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ng7t\" (UniqueName: \"kubernetes.io/projected/5f23f75f-fd4c-4592-84d5-ef0a20be658e-kube-api-access-4ng7t\") pod \"maas-api-revoke-sub-555cb312-wxk9t\" (UID: \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:23.066658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.066477 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555cb312-wxk9t\" (UID: \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:23.067270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.067238 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555cb312-wxk9t\" (UID: \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:23.068837 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.068805 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555cb312-wxk9t\" (UID: \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:23.090895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.090853 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ng7t\" (UniqueName: \"kubernetes.io/projected/5f23f75f-fd4c-4592-84d5-ef0a20be658e-kube-api-access-4ng7t\") pod \"maas-api-revoke-sub-555cb312-wxk9t\" (UID: \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:23.129907 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.129859 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:23.449999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.441013 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t"] Oct 30 18:10:23.449999 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:10:23.442420 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f23f75f_fd4c_4592_84d5_ef0a20be658e.slice/crio-2749cfd85c3e33af51eeae018a6daa48691a2f529f1e3475502b8c9c4302fdc3 WatchSource:0}: Error finding container 2749cfd85c3e33af51eeae018a6daa48691a2f529f1e3475502b8c9c4302fdc3: Status 404 returned error can't find the container with id 2749cfd85c3e33af51eeae018a6daa48691a2f529f1e3475502b8c9c4302fdc3 Oct 30 18:10:23.529303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.529263 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd"] Oct 30 18:10:23.534752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.534725 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:23.555403 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.555362 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd"] Oct 30 18:10:23.678152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.677206 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h87x\" (UniqueName: \"kubernetes.io/projected/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-kube-api-access-7h87x\") pod \"maas-api-revoke-sub-74ca3714-2kjjd\" (UID: \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:23.678152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.677344 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74ca3714-2kjjd\" (UID: \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:23.678152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.677408 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74ca3714-2kjjd\" (UID: \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:23.756059 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.755964 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" event={"ID":"5f23f75f-fd4c-4592-84d5-ef0a20be658e","Type":"ContainerStarted","Data":"2749cfd85c3e33af51eeae018a6daa48691a2f529f1e3475502b8c9c4302fdc3"} Oct 30 18:10:23.778427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.778395 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74ca3714-2kjjd\" (UID: \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:23.778528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.778489 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74ca3714-2kjjd\" (UID: \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:23.778593 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.778573 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h87x\" (UniqueName: \"kubernetes.io/projected/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-kube-api-access-7h87x\") pod \"maas-api-revoke-sub-74ca3714-2kjjd\" (UID: \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:23.779278 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.779254 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74ca3714-2kjjd\" (UID: \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:23.780810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.780783 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74ca3714-2kjjd\" (UID: \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:23.803327 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.803292 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h87x\" (UniqueName: \"kubernetes.io/projected/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-kube-api-access-7h87x\") pod \"maas-api-revoke-sub-74ca3714-2kjjd\" (UID: \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:23.849156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:23.849117 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:24.189893 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:24.189860 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd"] Oct 30 18:10:24.190562 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:10:24.190532 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2b59af_517b_4b0b_aaf4_0d8c77a3f7a4.slice/crio-078bac93a08612ffe60ab3e17474bc0b186b16df8f66df500d67522237355874 WatchSource:0}: Error finding container 078bac93a08612ffe60ab3e17474bc0b186b16df8f66df500d67522237355874: Status 404 returned error can't find the container with id 078bac93a08612ffe60ab3e17474bc0b186b16df8f66df500d67522237355874 Oct 30 18:10:24.766147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:24.766039 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" event={"ID":"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4","Type":"ContainerStarted","Data":"ce590409d8d7286a892ae3a10e714a2793cc9cd577158ea26f3f3978c0cda58a"} Oct 30 18:10:24.766147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:24.766119 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" event={"ID":"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4","Type":"ContainerStarted","Data":"078bac93a08612ffe60ab3e17474bc0b186b16df8f66df500d67522237355874"} Oct 30 18:10:24.768181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:24.768135 4883 generic.go:356] "Generic (PLEG): container finished" podID="5f23f75f-fd4c-4592-84d5-ef0a20be658e" containerID="e4733297d180f132ceb715f4f598fde4b4e6f4a962393eb64a9a7a0ca3a735b3" exitCode=0 Oct 30 18:10:24.768324 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:24.768249 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" event={"ID":"5f23f75f-fd4c-4592-84d5-ef0a20be658e","Type":"ContainerDied","Data":"e4733297d180f132ceb715f4f598fde4b4e6f4a962393eb64a9a7a0ca3a735b3"} Oct 30 18:10:24.808892 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:24.808655 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" podStartSLOduration=1.8086383929999998 podStartE2EDuration="1.808638393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:24.807610241 +0000 UTC m=+1083.764614076" watchObservedRunningTime="2026-10-30 18:10:24.808638393 +0000 UTC m=+1083.765642229" Oct 30 18:10:25.776719 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:25.776682 4883 generic.go:356] "Generic (PLEG): container finished" podID="2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4" containerID="ce590409d8d7286a892ae3a10e714a2793cc9cd577158ea26f3f3978c0cda58a" exitCode=0 Oct 30 18:10:25.777172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:25.776766 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" event={"ID":"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4","Type":"ContainerDied","Data":"ce590409d8d7286a892ae3a10e714a2793cc9cd577158ea26f3f3978c0cda58a"} Oct 30 18:10:25.953850 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:25.953818 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:26.108887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.108847 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-openshift-ca\") pod \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\" (UID: \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\") " Oct 30 18:10:26.109125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.109067 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f23f75f-fd4c-4592-84d5-ef0a20be658e-kube-api-access-4ng7t\" (UniqueName: \"kubernetes.io/projected/5f23f75f-fd4c-4592-84d5-ef0a20be658e-kube-api-access-4ng7t\") pod \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\" (UID: \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\") " Oct 30 18:10:26.109219 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.109202 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-xks-ca\") pod \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\" (UID: \"5f23f75f-fd4c-4592-84d5-ef0a20be658e\") " Oct 30 18:10:26.109315 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.109287 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-openshift-ca" pod "5f23f75f-fd4c-4592-84d5-ef0a20be658e" (UID: "5f23f75f-fd4c-4592-84d5-ef0a20be658e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:26.109845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.109823 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:26.111272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.111246 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f23f75f-fd4c-4592-84d5-ef0a20be658e-kube-api-access-4ng7t" pod "5f23f75f-fd4c-4592-84d5-ef0a20be658e" (UID: "5f23f75f-fd4c-4592-84d5-ef0a20be658e"). InnerVolumeSpecName "kube-api-access-4ng7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:26.111410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.111387 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-xks-ca" pod "5f23f75f-fd4c-4592-84d5-ef0a20be658e" (UID: "5f23f75f-fd4c-4592-84d5-ef0a20be658e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:26.211713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.211672 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ng7t\" (UniqueName: \"kubernetes.io/projected/5f23f75f-fd4c-4592-84d5-ef0a20be658e-kube-api-access-4ng7t\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:26.211713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.211709 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f23f75f-fd4c-4592-84d5-ef0a20be658e-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:26.670841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.670799 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv"] Oct 30 18:10:26.671610 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.671590 4883 cpu_manager.go:414] "containerMap: removing container" podUID="5f23f75f-fd4c-4592-84d5-ef0a20be658e" containerName="revoke-keys" Oct 30 18:10:26.671663 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.671611 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f23f75f-fd4c-4592-84d5-ef0a20be658e" containerName="revoke-keys" Oct 30 18:10:26.671727 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.671718 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f23f75f-fd4c-4592-84d5-ef0a20be658e" containerName="revoke-keys" Oct 30 18:10:26.675509 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.675490 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:26.697097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.697059 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv"] Oct 30 18:10:26.785808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.785776 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" Oct 30 18:10:26.786291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.785771 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t" event={"ID":"5f23f75f-fd4c-4592-84d5-ef0a20be658e","Type":"ContainerDied","Data":"2749cfd85c3e33af51eeae018a6daa48691a2f529f1e3475502b8c9c4302fdc3"} Oct 30 18:10:26.786291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.785921 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2749cfd85c3e33af51eeae018a6daa48691a2f529f1e3475502b8c9c4302fdc3" Oct 30 18:10:26.819538 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.819415 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c262x\" (UniqueName: \"kubernetes.io/projected/77e2118c-04b2-4055-b486-a5a15f981af9-kube-api-access-c262x\") pod \"maas-api-revoke-sub-bb5547d1-d7mjv\" (UID: \"77e2118c-04b2-4055-b486-a5a15f981af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:26.824772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.823100 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb5547d1-d7mjv\" (UID: \"77e2118c-04b2-4055-b486-a5a15f981af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:26.824772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.823356 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb5547d1-d7mjv\" (UID: \"77e2118c-04b2-4055-b486-a5a15f981af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:26.924457 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.924361 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c262x\" (UniqueName: \"kubernetes.io/projected/77e2118c-04b2-4055-b486-a5a15f981af9-kube-api-access-c262x\") pod \"maas-api-revoke-sub-bb5547d1-d7mjv\" (UID: \"77e2118c-04b2-4055-b486-a5a15f981af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:26.924622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.924457 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb5547d1-d7mjv\" (UID: \"77e2118c-04b2-4055-b486-a5a15f981af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:26.924694 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.924626 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb5547d1-d7mjv\" (UID: \"77e2118c-04b2-4055-b486-a5a15f981af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:26.925201 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.925163 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb5547d1-d7mjv\" (UID: \"77e2118c-04b2-4055-b486-a5a15f981af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:26.927998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.927965 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb5547d1-d7mjv\" (UID: \"77e2118c-04b2-4055-b486-a5a15f981af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:26.949107 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.949075 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c262x\" (UniqueName: \"kubernetes.io/projected/77e2118c-04b2-4055-b486-a5a15f981af9-kube-api-access-c262x\") pod \"maas-api-revoke-sub-bb5547d1-d7mjv\" (UID: \"77e2118c-04b2-4055-b486-a5a15f981af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:26.977051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.977023 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:26.990617 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:26.990581 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:27.127601 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.127560 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-openshift-ca\") pod \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\" (UID: \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\") " Oct 30 18:10:27.127816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.127612 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-kube-api-access-7h87x\" (UniqueName: \"kubernetes.io/projected/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-kube-api-access-7h87x\") pod \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\" (UID: \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\") " Oct 30 18:10:27.127903 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.127880 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-xks-ca\") pod \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\" (UID: \"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4\") " Oct 30 18:10:27.128068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.128040 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-openshift-ca" pod "2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4" (UID: "2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:27.128886 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.128857 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:27.131546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.131510 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-xks-ca" pod "2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4" (UID: "2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:27.132321 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.132261 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-kube-api-access-7h87x" pod "2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4" (UID: "2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4"). InnerVolumeSpecName "kube-api-access-7h87x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:27.230524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.230487 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h87x\" (UniqueName: \"kubernetes.io/projected/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-kube-api-access-7h87x\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:27.230524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.230526 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:27.307579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.307548 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv"] Oct 30 18:10:27.308611 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:10:27.308583 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e2118c_04b2_4055_b486_a5a15f981af9.slice/crio-dbecae4898b9ce6ad7d69a1732d15fc315041bd575581e0ca2bdbe117651259d WatchSource:0}: Error finding container dbecae4898b9ce6ad7d69a1732d15fc315041bd575581e0ca2bdbe117651259d: Status 404 returned error can't find the container with id dbecae4898b9ce6ad7d69a1732d15fc315041bd575581e0ca2bdbe117651259d Oct 30 18:10:27.794902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.794822 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" Oct 30 18:10:27.795397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.794821 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd" event={"ID":"2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4","Type":"ContainerDied","Data":"078bac93a08612ffe60ab3e17474bc0b186b16df8f66df500d67522237355874"} Oct 30 18:10:27.795397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.794998 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="078bac93a08612ffe60ab3e17474bc0b186b16df8f66df500d67522237355874" Oct 30 18:10:27.796732 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.796705 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" event={"ID":"77e2118c-04b2-4055-b486-a5a15f981af9","Type":"ContainerStarted","Data":"fc40075c0672c969b876215668a350cc5fb2bf4de34663b1323a03b1ce152604"} Oct 30 18:10:27.796899 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.796753 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" event={"ID":"77e2118c-04b2-4055-b486-a5a15f981af9","Type":"ContainerStarted","Data":"dbecae4898b9ce6ad7d69a1732d15fc315041bd575581e0ca2bdbe117651259d"} Oct 30 18:10:27.839445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:27.839386 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" podStartSLOduration=1.839367941 podStartE2EDuration="1.839367941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:27.838695831 +0000 UTC m=+1086.795699685" watchObservedRunningTime="2026-10-30 18:10:27.839367941 +0000 UTC m=+1086.796371774" Oct 30 18:10:28.809269 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:28.809197 4883 generic.go:356] "Generic (PLEG): container finished" podID="77e2118c-04b2-4055-b486-a5a15f981af9" containerID="fc40075c0672c969b876215668a350cc5fb2bf4de34663b1323a03b1ce152604" exitCode=0 Oct 30 18:10:28.809784 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:28.809433 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" event={"ID":"77e2118c-04b2-4055-b486-a5a15f981af9","Type":"ContainerDied","Data":"fc40075c0672c969b876215668a350cc5fb2bf4de34663b1323a03b1ce152604"} Oct 30 18:10:29.982253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:29.982221 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:30.170704 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:30.170656 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-xks-ca\") pod \"77e2118c-04b2-4055-b486-a5a15f981af9\" (UID: \"77e2118c-04b2-4055-b486-a5a15f981af9\") " Oct 30 18:10:30.170908 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:30.170782 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e2118c-04b2-4055-b486-a5a15f981af9-kube-api-access-c262x\" (UniqueName: \"kubernetes.io/projected/77e2118c-04b2-4055-b486-a5a15f981af9-kube-api-access-c262x\") pod \"77e2118c-04b2-4055-b486-a5a15f981af9\" (UID: \"77e2118c-04b2-4055-b486-a5a15f981af9\") " Oct 30 18:10:30.171040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:30.170980 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-openshift-ca\") pod \"77e2118c-04b2-4055-b486-a5a15f981af9\" (UID: \"77e2118c-04b2-4055-b486-a5a15f981af9\") " Oct 30 18:10:30.171316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:30.171291 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-openshift-ca" pod "77e2118c-04b2-4055-b486-a5a15f981af9" (UID: "77e2118c-04b2-4055-b486-a5a15f981af9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:30.172950 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:30.172899 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-xks-ca" pod "77e2118c-04b2-4055-b486-a5a15f981af9" (UID: "77e2118c-04b2-4055-b486-a5a15f981af9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:30.172950 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:30.172931 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e2118c-04b2-4055-b486-a5a15f981af9-kube-api-access-c262x" pod "77e2118c-04b2-4055-b486-a5a15f981af9" (UID: "77e2118c-04b2-4055-b486-a5a15f981af9"). InnerVolumeSpecName "kube-api-access-c262x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:30.273111 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:30.273069 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:30.273111 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:30.273108 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e2118c-04b2-4055-b486-a5a15f981af9-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:30.273385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:30.273124 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c262x\" (UniqueName: \"kubernetes.io/projected/77e2118c-04b2-4055-b486-a5a15f981af9-kube-api-access-c262x\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:30.850971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:30.844605 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" event={"ID":"77e2118c-04b2-4055-b486-a5a15f981af9","Type":"ContainerDied","Data":"dbecae4898b9ce6ad7d69a1732d15fc315041bd575581e0ca2bdbe117651259d"} Oct 30 18:10:30.850971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:30.844708 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbecae4898b9ce6ad7d69a1732d15fc315041bd575581e0ca2bdbe117651259d" Oct 30 18:10:30.850971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:30.844776 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv" Oct 30 18:10:32.846960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:32.846113 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:32.847602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:32.847206 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:32.897164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:32.893796 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.001085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.001033 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.089805 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.089080 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.215019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.198070 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.416167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.416106 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.436302 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.436256 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.500150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.500038 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.568092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.568052 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:33.568396 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.568371 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.698086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.698035 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:10:33.853980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.847261 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:10:33.853980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.848065 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:10:33.853980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.849659 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.864897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.854787 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:10:33.916842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.908066 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:33.916842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:33.908827 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:10:34.112407 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.089427 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.112407 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.099060 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:34.112407 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.099126 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.183184 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.133143 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:10:34.210085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.203033 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.238962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.214357 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:10:34.398123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.398027 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:10:34.450961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.428596 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.623408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.452033 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.628960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.559081 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:10:34.710016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.629180 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:10:34.710016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.681355 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:10:34.710016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.681397 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:10:34.710016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.700081 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:34.710016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.700483 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:10:34.859113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.859040 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:35.028373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.886597 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:10:35.048445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:35.040720 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:10:35.063156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:34.936459 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:35.063485 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:35.063434 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:35.187963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:35.155172 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:10:35.291591 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:35.291482 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:35.521032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:35.520981 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:35.580145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:35.580034 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:35.629318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:35.628732 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:35.644958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:35.640963 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:10:35.691966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:35.691558 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:10:35.738095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:35.714980 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:10:35.742961 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:10:35.740474 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:10:35.765961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:35.754053 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:35.959361 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:35.959325 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:10:36.017749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:36.017715 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:10:36.029746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:36.029701 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:36.045753 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:36.045114 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:36.048964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:36.046250 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:10:36.374795 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:36.374747 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:10:36.537227 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:36.537159 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:36.538074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:36.538048 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:10:36.541915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:36.541873 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:10:36.650150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:36.650035 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:36.674531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:36.674484 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:36.896103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:36.896051 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.032022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:37.025809 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.032624 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:37.032586 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:10:37.057962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:37.053018 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:10:37.380713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:37.380666 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:10:37.538808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:37.537670 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.543259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:37.542455 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.570679 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:37.569186 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:10:37.665019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:37.662106 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.682974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:37.680029 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:37.774599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:37.774548 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.034858 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.034742 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.120171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.087146 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.120171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.088270 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.120171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.091816 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.120171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.119297 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:10:38.261347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.261310 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:38.261618 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.261587 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.403961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.399457 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.436608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.436572 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92"] Oct 30 18:10:38.447951 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.447904 4883 cpu_manager.go:414] "containerMap: removing container" podUID="2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4" containerName="revoke-keys" Oct 30 18:10:38.523333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.520036 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4" containerName="revoke-keys" Oct 30 18:10:38.523333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.520095 4883 cpu_manager.go:414] "containerMap: removing container" podUID="77e2118c-04b2-4055-b486-a5a15f981af9" containerName="revoke-keys" Oct 30 18:10:38.523333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.520105 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e2118c-04b2-4055-b486-a5a15f981af9" containerName="revoke-keys" Oct 30 18:10:38.523333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.520541 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4" containerName="revoke-keys" Oct 30 18:10:38.523333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.520565 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e2118c-04b2-4055-b486-a5a15f981af9" containerName="revoke-keys" Oct 30 18:10:38.534183 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.441308 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.537678 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.537585 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92"] Oct 30 18:10:38.573168 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.573045 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:38.579800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.579767 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:10:38.581999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.581974 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:38.583096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.583076 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:10:38.583267 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.583257 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:10:38.586765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.583371 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:10:38.586947 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.586921 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:10:38.609970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.599037 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:10:38.652276 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.652240 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:10:38.653337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.653304 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:10:38.653476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.653355 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:10:38.653476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.653429 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:10:38.744960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.733268 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:10:38.773390 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.773348 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ea3406-tds92\" (UID: \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:38.773763 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.773735 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ea3406-tds92\" (UID: \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:38.774219 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.774197 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wsm\" (UniqueName: \"kubernetes.io/projected/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-kube-api-access-47wsm\") pod \"maas-api-revoke-sub-a1ea3406-tds92\" (UID: \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:38.877081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.875296 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ea3406-tds92\" (UID: \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:38.877081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.875379 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ea3406-tds92\" (UID: \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:38.877081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.875517 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wsm\" (UniqueName: \"kubernetes.io/projected/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-kube-api-access-47wsm\") pod \"maas-api-revoke-sub-a1ea3406-tds92\" (UID: \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:38.877081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.876711 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ea3406-tds92\" (UID: \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:38.881657 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.881622 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ea3406-tds92\" (UID: \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:38.910566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:38.910529 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wsm\" (UniqueName: \"kubernetes.io/projected/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-kube-api-access-47wsm\") pod \"maas-api-revoke-sub-a1ea3406-tds92\" (UID: \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:39.023627 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:39.023537 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:39.340547 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:39.340511 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92"] Oct 30 18:10:39.341686 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:10:39.341654 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8cb9c9_a632_4ffa_8a9b_9c2eb1ea6c6c.slice/crio-ab915a726c28a29d81e2e9ff36fb6b7d7a65b54ae9e169e7853dac76fbf42c2e WatchSource:0}: Error finding container ab915a726c28a29d81e2e9ff36fb6b7d7a65b54ae9e169e7853dac76fbf42c2e: Status 404 returned error can't find the container with id ab915a726c28a29d81e2e9ff36fb6b7d7a65b54ae9e169e7853dac76fbf42c2e Oct 30 18:10:40.211069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:40.211032 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" event={"ID":"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c","Type":"ContainerStarted","Data":"cb3a9b34912a40b2d5953515c0ad746cf166178026bfa18800a502c15205036b"} Oct 30 18:10:40.211265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:40.211096 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" event={"ID":"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c","Type":"ContainerStarted","Data":"ab915a726c28a29d81e2e9ff36fb6b7d7a65b54ae9e169e7853dac76fbf42c2e"} Oct 30 18:10:40.253922 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:40.253852 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" podStartSLOduration=3.253797428 podStartE2EDuration="3.253797428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:40.25309643 +0000 UTC m=+1099.210100266" watchObservedRunningTime="2026-10-30 18:10:40.253797428 +0000 UTC m=+1099.210801304" Oct 30 18:10:41.224326 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:41.224290 4883 generic.go:356] "Generic (PLEG): container finished" podID="9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c" containerID="cb3a9b34912a40b2d5953515c0ad746cf166178026bfa18800a502c15205036b" exitCode=0 Oct 30 18:10:41.224838 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:41.224381 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" event={"ID":"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c","Type":"ContainerDied","Data":"cb3a9b34912a40b2d5953515c0ad746cf166178026bfa18800a502c15205036b"} Oct 30 18:10:42.398554 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:42.398512 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:42.419973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:42.419895 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-xks-ca\") pod \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\" (UID: \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\") " Oct 30 18:10:42.420148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:42.420099 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-openshift-ca\") pod \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\" (UID: \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\") " Oct 30 18:10:42.420148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:42.420141 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-kube-api-access-47wsm\" (UniqueName: \"kubernetes.io/projected/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-kube-api-access-47wsm\") pod \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\" (UID: \"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c\") " Oct 30 18:10:42.420538 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:42.420499 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-openshift-ca" pod "9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c" (UID: "9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:42.422431 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:42.422402 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-xks-ca" pod "9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c" (UID: "9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:42.422552 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:42.422402 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-kube-api-access-47wsm" pod "9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c" (UID: "9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c"). InnerVolumeSpecName "kube-api-access-47wsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:42.521538 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:42.521438 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:42.521538 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:42.521479 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:42.521538 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:42.521494 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47wsm\" (UniqueName: \"kubernetes.io/projected/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c-kube-api-access-47wsm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:43.030647 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.030609 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl"] Oct 30 18:10:43.031576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.031554 4883 cpu_manager.go:414] "containerMap: removing container" podUID="9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c" containerName="revoke-keys" Oct 30 18:10:43.031654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.031579 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c" containerName="revoke-keys" Oct 30 18:10:43.031743 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.031731 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c" containerName="revoke-keys" Oct 30 18:10:43.039249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.039223 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:43.057015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.056980 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl"] Oct 30 18:10:43.129039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.128993 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0af5393-clgnl\" (UID: \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:43.129039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.129038 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0af5393-clgnl\" (UID: \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:43.129300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.129240 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k6xf\" (UniqueName: \"kubernetes.io/projected/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-kube-api-access-9k6xf\") pod \"maas-api-revoke-sub-a0af5393-clgnl\" (UID: \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:43.230010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.229967 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6xf\" (UniqueName: \"kubernetes.io/projected/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-kube-api-access-9k6xf\") pod \"maas-api-revoke-sub-a0af5393-clgnl\" (UID: \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:43.230214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.230154 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0af5393-clgnl\" (UID: \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:43.230214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.230196 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0af5393-clgnl\" (UID: \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:43.230998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.230972 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0af5393-clgnl\" (UID: \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:43.232879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.232855 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0af5393-clgnl\" (UID: \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:43.242955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.242918 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" Oct 30 18:10:43.243428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.242958 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92" event={"ID":"9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c","Type":"ContainerDied","Data":"ab915a726c28a29d81e2e9ff36fb6b7d7a65b54ae9e169e7853dac76fbf42c2e"} Oct 30 18:10:43.243553 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.243453 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab915a726c28a29d81e2e9ff36fb6b7d7a65b54ae9e169e7853dac76fbf42c2e" Oct 30 18:10:43.254535 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.254507 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k6xf\" (UniqueName: \"kubernetes.io/projected/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-kube-api-access-9k6xf\") pod \"maas-api-revoke-sub-a0af5393-clgnl\" (UID: \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:43.354628 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.354599 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:43.689610 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:43.684703 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl"] Oct 30 18:10:44.253155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:44.253062 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" event={"ID":"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795","Type":"ContainerStarted","Data":"b9f1515a31b6f6b7c25e0eac6cf4bbd72a2c84c54b53ea0b5cb77241eb68ff01"} Oct 30 18:10:44.253155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:44.253118 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" event={"ID":"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795","Type":"ContainerStarted","Data":"19f4e555bce5fd2bf4b042c0965516d74476387c56753b2808ffe6c939d53409"} Oct 30 18:10:44.300317 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:44.300256 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" podStartSLOduration=1.300238749 podStartE2EDuration="1.300238749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:44.296244628 +0000 UTC m=+1103.253248468" watchObservedRunningTime="2026-10-30 18:10:44.300238749 +0000 UTC m=+1103.257242627" Oct 30 18:10:45.267125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:45.267066 4883 generic.go:356] "Generic (PLEG): container finished" podID="b4ce6cf1-12ff-4a5b-b7b9-36527ef10795" containerID="b9f1515a31b6f6b7c25e0eac6cf4bbd72a2c84c54b53ea0b5cb77241eb68ff01" exitCode=0 Oct 30 18:10:45.267623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:45.267263 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" event={"ID":"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795","Type":"ContainerDied","Data":"b9f1515a31b6f6b7c25e0eac6cf4bbd72a2c84c54b53ea0b5cb77241eb68ff01"} Oct 30 18:10:46.449665 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:46.449630 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:46.579612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:46.579513 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-kube-api-access-9k6xf\" (UniqueName: \"kubernetes.io/projected/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-kube-api-access-9k6xf\") pod \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\" (UID: \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\") " Oct 30 18:10:46.579774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:46.579626 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-openshift-ca\") pod \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\" (UID: \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\") " Oct 30 18:10:46.579825 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:46.579787 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-xks-ca\") pod \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\" (UID: \"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795\") " Oct 30 18:10:46.580112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:46.580079 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-openshift-ca" pod "b4ce6cf1-12ff-4a5b-b7b9-36527ef10795" (UID: "b4ce6cf1-12ff-4a5b-b7b9-36527ef10795"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:46.580608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:46.580576 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:46.582056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:46.582025 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-xks-ca" pod "b4ce6cf1-12ff-4a5b-b7b9-36527ef10795" (UID: "b4ce6cf1-12ff-4a5b-b7b9-36527ef10795"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:46.582258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:46.582240 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-kube-api-access-9k6xf" pod "b4ce6cf1-12ff-4a5b-b7b9-36527ef10795" (UID: "b4ce6cf1-12ff-4a5b-b7b9-36527ef10795"). InnerVolumeSpecName "kube-api-access-9k6xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:46.681531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:46.681487 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k6xf\" (UniqueName: \"kubernetes.io/projected/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-kube-api-access-9k6xf\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:46.681531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:46.681523 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:47.282374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:47.282347 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" Oct 30 18:10:47.282587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:47.282339 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl" event={"ID":"b4ce6cf1-12ff-4a5b-b7b9-36527ef10795","Type":"ContainerDied","Data":"19f4e555bce5fd2bf4b042c0965516d74476387c56753b2808ffe6c939d53409"} Oct 30 18:10:47.282587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:47.282476 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19f4e555bce5fd2bf4b042c0965516d74476387c56753b2808ffe6c939d53409" Oct 30 18:10:49.012804 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.012767 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w"] Oct 30 18:10:49.013478 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.013459 4883 cpu_manager.go:414] "containerMap: removing container" podUID="b4ce6cf1-12ff-4a5b-b7b9-36527ef10795" containerName="revoke-keys" Oct 30 18:10:49.013533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.013480 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ce6cf1-12ff-4a5b-b7b9-36527ef10795" containerName="revoke-keys" Oct 30 18:10:49.013608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.013599 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ce6cf1-12ff-4a5b-b7b9-36527ef10795" containerName="revoke-keys" Oct 30 18:10:49.017830 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.017804 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:49.039235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.039199 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w"] Oct 30 18:10:49.047081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.047040 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:10:49.047779 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.047757 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:10:49.047928 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.047839 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:10:49.047928 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.047897 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:10:49.118498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.118454 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4sxd\" (UniqueName: \"kubernetes.io/projected/a3e4a4d5-a678-4deb-9639-221db9eca09a-kube-api-access-j4sxd\") pod \"maas-api-revoke-sub-b8fc3315-zvq4w\" (UID: \"a3e4a4d5-a678-4deb-9639-221db9eca09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:49.118669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.118633 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8fc3315-zvq4w\" (UID: \"a3e4a4d5-a678-4deb-9639-221db9eca09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:49.118732 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.118690 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8fc3315-zvq4w\" (UID: \"a3e4a4d5-a678-4deb-9639-221db9eca09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:49.219611 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.219569 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4sxd\" (UniqueName: \"kubernetes.io/projected/a3e4a4d5-a678-4deb-9639-221db9eca09a-kube-api-access-j4sxd\") pod \"maas-api-revoke-sub-b8fc3315-zvq4w\" (UID: \"a3e4a4d5-a678-4deb-9639-221db9eca09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:49.219796 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.219680 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8fc3315-zvq4w\" (UID: \"a3e4a4d5-a678-4deb-9639-221db9eca09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:49.219796 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.219714 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8fc3315-zvq4w\" (UID: \"a3e4a4d5-a678-4deb-9639-221db9eca09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:49.220520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.220492 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8fc3315-zvq4w\" (UID: \"a3e4a4d5-a678-4deb-9639-221db9eca09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:49.222229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.222198 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8fc3315-zvq4w\" (UID: \"a3e4a4d5-a678-4deb-9639-221db9eca09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:49.244284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.244252 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4sxd\" (UniqueName: \"kubernetes.io/projected/a3e4a4d5-a678-4deb-9639-221db9eca09a-kube-api-access-j4sxd\") pod \"maas-api-revoke-sub-b8fc3315-zvq4w\" (UID: \"a3e4a4d5-a678-4deb-9639-221db9eca09a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:49.333364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.333254 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:49.644118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:49.644084 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w"] Oct 30 18:10:49.644471 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:10:49.644443 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e4a4d5_a678_4deb_9639_221db9eca09a.slice/crio-abaccedc9c138b7e3c048c08205fcfa26157c8f2d27025162628eea08c534102 WatchSource:0}: Error finding container abaccedc9c138b7e3c048c08205fcfa26157c8f2d27025162628eea08c534102: Status 404 returned error can't find the container with id abaccedc9c138b7e3c048c08205fcfa26157c8f2d27025162628eea08c534102 Oct 30 18:10:50.307766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:50.307660 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" event={"ID":"a3e4a4d5-a678-4deb-9639-221db9eca09a","Type":"ContainerStarted","Data":"b47722d1ffca4f38345756cce58dfe58b2ac59b13f5dd8795f8b70b084c74710"} Oct 30 18:10:50.307766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:50.307722 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" event={"ID":"a3e4a4d5-a678-4deb-9639-221db9eca09a","Type":"ContainerStarted","Data":"abaccedc9c138b7e3c048c08205fcfa26157c8f2d27025162628eea08c534102"} Oct 30 18:10:51.316474 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:51.316207 4883 generic.go:356] "Generic (PLEG): container finished" podID="a3e4a4d5-a678-4deb-9639-221db9eca09a" containerID="b47722d1ffca4f38345756cce58dfe58b2ac59b13f5dd8795f8b70b084c74710" exitCode=0 Oct 30 18:10:51.316992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:51.316292 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" event={"ID":"a3e4a4d5-a678-4deb-9639-221db9eca09a","Type":"ContainerDied","Data":"b47722d1ffca4f38345756cce58dfe58b2ac59b13f5dd8795f8b70b084c74710"} Oct 30 18:10:52.506420 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:52.506390 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:52.565658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:52.565625 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-xks-ca\") pod \"a3e4a4d5-a678-4deb-9639-221db9eca09a\" (UID: \"a3e4a4d5-a678-4deb-9639-221db9eca09a\") " Oct 30 18:10:52.565826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:52.565682 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-openshift-ca\") pod \"a3e4a4d5-a678-4deb-9639-221db9eca09a\" (UID: \"a3e4a4d5-a678-4deb-9639-221db9eca09a\") " Oct 30 18:10:52.565826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:52.565771 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e4a4d5-a678-4deb-9639-221db9eca09a-kube-api-access-j4sxd\" (UniqueName: \"kubernetes.io/projected/a3e4a4d5-a678-4deb-9639-221db9eca09a-kube-api-access-j4sxd\") pod \"a3e4a4d5-a678-4deb-9639-221db9eca09a\" (UID: \"a3e4a4d5-a678-4deb-9639-221db9eca09a\") " Oct 30 18:10:52.566189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:52.566159 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-openshift-ca" pod "a3e4a4d5-a678-4deb-9639-221db9eca09a" (UID: "a3e4a4d5-a678-4deb-9639-221db9eca09a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:52.566414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:52.566338 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:52.568774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:52.568743 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e4a4d5-a678-4deb-9639-221db9eca09a-kube-api-access-j4sxd" pod "a3e4a4d5-a678-4deb-9639-221db9eca09a" (UID: "a3e4a4d5-a678-4deb-9639-221db9eca09a"). InnerVolumeSpecName "kube-api-access-j4sxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:52.571259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:52.571227 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-xks-ca" pod "a3e4a4d5-a678-4deb-9639-221db9eca09a" (UID: "a3e4a4d5-a678-4deb-9639-221db9eca09a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:10:52.668327 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:52.668293 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e4a4d5-a678-4deb-9639-221db9eca09a-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:52.668327 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:52.668326 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4sxd\" (UniqueName: \"kubernetes.io/projected/a3e4a4d5-a678-4deb-9639-221db9eca09a-kube-api-access-j4sxd\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:10:53.333116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:53.333082 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" event={"ID":"a3e4a4d5-a678-4deb-9639-221db9eca09a","Type":"ContainerDied","Data":"abaccedc9c138b7e3c048c08205fcfa26157c8f2d27025162628eea08c534102"} Oct 30 18:10:53.333283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:53.333136 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abaccedc9c138b7e3c048c08205fcfa26157c8f2d27025162628eea08c534102" Oct 30 18:10:53.333283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:53.333093 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w" Oct 30 18:10:59.041794 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.041751 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz"] Oct 30 18:10:59.057242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.057207 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad529b6e-zk8jz"] Oct 30 18:10:59.178538 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.178502 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9444bfd-5672-413b-b80d-6b6bea511a7f" path="/var/lib/kubelet/pods/c9444bfd-5672-413b-b80d-6b6bea511a7f/volumes" Oct 30 18:10:59.340341 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.340259 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk"] Oct 30 18:10:59.340823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.340808 4883 cpu_manager.go:414] "containerMap: removing container" podUID="a3e4a4d5-a678-4deb-9639-221db9eca09a" containerName="revoke-keys" Oct 30 18:10:59.340866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.340823 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e4a4d5-a678-4deb-9639-221db9eca09a" containerName="revoke-keys" Oct 30 18:10:59.340968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.340928 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e4a4d5-a678-4deb-9639-221db9eca09a" containerName="revoke-keys" Oct 30 18:10:59.345880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.345858 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:10:59.368118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.368089 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:10:59.368325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.368190 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:10:59.368325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.368236 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:10:59.368325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.368252 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:10:59.368325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.368302 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk"] Oct 30 18:10:59.464014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.463967 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cfb2fdd-gxklk\" (UID: \"8632419b-f939-49f2-854a-a1b3903d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:10:59.464184 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.464042 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cfb2fdd-gxklk\" (UID: \"8632419b-f939-49f2-854a-a1b3903d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:10:59.464266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.464178 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktr6x\" (UniqueName: \"kubernetes.io/projected/8632419b-f939-49f2-854a-a1b3903d8560-kube-api-access-ktr6x\") pod \"maas-api-revoke-sub-0cfb2fdd-gxklk\" (UID: \"8632419b-f939-49f2-854a-a1b3903d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:10:59.565380 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.565338 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cfb2fdd-gxklk\" (UID: \"8632419b-f939-49f2-854a-a1b3903d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:10:59.565586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.565402 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cfb2fdd-gxklk\" (UID: \"8632419b-f939-49f2-854a-a1b3903d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:10:59.565586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.565444 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktr6x\" (UniqueName: \"kubernetes.io/projected/8632419b-f939-49f2-854a-a1b3903d8560-kube-api-access-ktr6x\") pod \"maas-api-revoke-sub-0cfb2fdd-gxklk\" (UID: \"8632419b-f939-49f2-854a-a1b3903d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:10:59.566189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.566165 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cfb2fdd-gxklk\" (UID: \"8632419b-f939-49f2-854a-a1b3903d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:10:59.567975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.567903 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cfb2fdd-gxklk\" (UID: \"8632419b-f939-49f2-854a-a1b3903d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:10:59.590796 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.590721 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktr6x\" (UniqueName: \"kubernetes.io/projected/8632419b-f939-49f2-854a-a1b3903d8560-kube-api-access-ktr6x\") pod \"maas-api-revoke-sub-0cfb2fdd-gxklk\" (UID: \"8632419b-f939-49f2-854a-a1b3903d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:10:59.663519 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.663480 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:10:59.979761 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:10:59.979728 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk"] Oct 30 18:10:59.979889 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:10:59.979825 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8632419b_f939_49f2_854a_a1b3903d8560.slice/crio-466abe311ab86e2d45c8bd5d38048e592d34855a7c83498d2e1323705aa8d6bc WatchSource:0}: Error finding container 466abe311ab86e2d45c8bd5d38048e592d34855a7c83498d2e1323705aa8d6bc: Status 404 returned error can't find the container with id 466abe311ab86e2d45c8bd5d38048e592d34855a7c83498d2e1323705aa8d6bc Oct 30 18:11:00.391697 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:00.391657 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" event={"ID":"8632419b-f939-49f2-854a-a1b3903d8560","Type":"ContainerStarted","Data":"ccf603365b7b48c6d7b4f8c7d9e60565a794d0165605ba8624c3ce98e418bb06"} Oct 30 18:11:00.392176 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:00.391720 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" event={"ID":"8632419b-f939-49f2-854a-a1b3903d8560","Type":"ContainerStarted","Data":"466abe311ab86e2d45c8bd5d38048e592d34855a7c83498d2e1323705aa8d6bc"} Oct 30 18:11:01.400714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:01.400675 4883 generic.go:356] "Generic (PLEG): container finished" podID="8632419b-f939-49f2-854a-a1b3903d8560" containerID="ccf603365b7b48c6d7b4f8c7d9e60565a794d0165605ba8624c3ce98e418bb06" exitCode=0 Oct 30 18:11:01.401168 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:01.400754 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" event={"ID":"8632419b-f939-49f2-854a-a1b3903d8560","Type":"ContainerDied","Data":"ccf603365b7b48c6d7b4f8c7d9e60565a794d0165605ba8624c3ce98e418bb06"} Oct 30 18:11:02.510528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.510481 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8"] Oct 30 18:11:02.525122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.521591 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:02.541993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.539610 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8"] Oct 30 18:11:02.619918 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.619885 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:11:02.709143 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.709048 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a984a3c4-7g9c8\" (UID: \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:02.709335 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.709297 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a984a3c4-7g9c8\" (UID: \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:02.709462 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.709426 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzqz\" (UniqueName: \"kubernetes.io/projected/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-kube-api-access-smzqz\") pod \"maas-api-revoke-sub-a984a3c4-7g9c8\" (UID: \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:02.810280 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.810239 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-xks-ca\") pod \"8632419b-f939-49f2-854a-a1b3903d8560\" (UID: \"8632419b-f939-49f2-854a-a1b3903d8560\") " Oct 30 18:11:02.810477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.810311 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8632419b-f939-49f2-854a-a1b3903d8560-kube-api-access-ktr6x\" (UniqueName: \"kubernetes.io/projected/8632419b-f939-49f2-854a-a1b3903d8560-kube-api-access-ktr6x\") pod \"8632419b-f939-49f2-854a-a1b3903d8560\" (UID: \"8632419b-f939-49f2-854a-a1b3903d8560\") " Oct 30 18:11:02.810477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.810403 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-openshift-ca\") pod \"8632419b-f939-49f2-854a-a1b3903d8560\" (UID: \"8632419b-f939-49f2-854a-a1b3903d8560\") " Oct 30 18:11:02.810758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.810732 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-openshift-ca" pod "8632419b-f939-49f2-854a-a1b3903d8560" (UID: "8632419b-f939-49f2-854a-a1b3903d8560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:02.810841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.810818 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a984a3c4-7g9c8\" (UID: \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:02.810985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.810966 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a984a3c4-7g9c8\" (UID: \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:02.811074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.811044 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzqz\" (UniqueName: \"kubernetes.io/projected/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-kube-api-access-smzqz\") pod \"maas-api-revoke-sub-a984a3c4-7g9c8\" (UID: \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:02.811235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.811220 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:02.811784 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.811761 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a984a3c4-7g9c8\" (UID: \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:02.812649 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.812620 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8632419b-f939-49f2-854a-a1b3903d8560-kube-api-access-ktr6x" pod "8632419b-f939-49f2-854a-a1b3903d8560" (UID: "8632419b-f939-49f2-854a-a1b3903d8560"). InnerVolumeSpecName "kube-api-access-ktr6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:02.812649 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.812635 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-xks-ca" pod "8632419b-f939-49f2-854a-a1b3903d8560" (UID: "8632419b-f939-49f2-854a-a1b3903d8560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:02.813124 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.813107 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a984a3c4-7g9c8\" (UID: \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:02.836035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.836003 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzqz\" (UniqueName: \"kubernetes.io/projected/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-kube-api-access-smzqz\") pod \"maas-api-revoke-sub-a984a3c4-7g9c8\" (UID: \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:02.900010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.899973 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:02.914456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.914420 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8632419b-f939-49f2-854a-a1b3903d8560-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:02.914456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:02.914455 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktr6x\" (UniqueName: \"kubernetes.io/projected/8632419b-f939-49f2-854a-a1b3903d8560-kube-api-access-ktr6x\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:03.215440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:03.215405 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8"] Oct 30 18:11:03.216060 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:11:03.216031 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1617a80d_af7d_4904_89cb_8c53cd8bd5b8.slice/crio-44b3aa08a4e4400591d6b82283a73bbac10005b9cebaa77cc35c8d56c5bfdd5c WatchSource:0}: Error finding container 44b3aa08a4e4400591d6b82283a73bbac10005b9cebaa77cc35c8d56c5bfdd5c: Status 404 returned error can't find the container with id 44b3aa08a4e4400591d6b82283a73bbac10005b9cebaa77cc35c8d56c5bfdd5c Oct 30 18:11:03.415835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:03.415800 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" Oct 30 18:11:03.415835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:03.415827 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk" event={"ID":"8632419b-f939-49f2-854a-a1b3903d8560","Type":"ContainerDied","Data":"466abe311ab86e2d45c8bd5d38048e592d34855a7c83498d2e1323705aa8d6bc"} Oct 30 18:11:03.416082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:03.415877 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="466abe311ab86e2d45c8bd5d38048e592d34855a7c83498d2e1323705aa8d6bc" Oct 30 18:11:03.417443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:03.417407 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" event={"ID":"1617a80d-af7d-4904-89cb-8c53cd8bd5b8","Type":"ContainerStarted","Data":"44b3aa08a4e4400591d6b82283a73bbac10005b9cebaa77cc35c8d56c5bfdd5c"} Oct 30 18:11:04.426367 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:04.426327 4883 generic.go:356] "Generic (PLEG): container finished" podID="1617a80d-af7d-4904-89cb-8c53cd8bd5b8" containerID="235e0079332b6288c51c97051ae2703aaba0398b1ca9c8c0aeccdb82c4fbc843" exitCode=0 Oct 30 18:11:04.426880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:04.426408 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" event={"ID":"1617a80d-af7d-4904-89cb-8c53cd8bd5b8","Type":"ContainerDied","Data":"235e0079332b6288c51c97051ae2703aaba0398b1ca9c8c0aeccdb82c4fbc843"} Oct 30 18:11:05.080892 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:11:05.080842 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:11:05.628509 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:05.628478 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:05.666173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:05.665133 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-openshift-ca\") pod \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\" (UID: \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\") " Oct 30 18:11:05.666173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:05.665546 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-openshift-ca" pod "1617a80d-af7d-4904-89cb-8c53cd8bd5b8" (UID: "1617a80d-af7d-4904-89cb-8c53cd8bd5b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:05.666173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:05.665859 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-xks-ca\") pod \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\" (UID: \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\") " Oct 30 18:11:05.666173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:05.665900 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-kube-api-access-smzqz\" (UniqueName: \"kubernetes.io/projected/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-kube-api-access-smzqz\") pod \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\" (UID: \"1617a80d-af7d-4904-89cb-8c53cd8bd5b8\") " Oct 30 18:11:05.666540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:05.666435 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:05.668618 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:05.668586 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-xks-ca" pod "1617a80d-af7d-4904-89cb-8c53cd8bd5b8" (UID: "1617a80d-af7d-4904-89cb-8c53cd8bd5b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:05.668618 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:05.668611 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-kube-api-access-smzqz" pod "1617a80d-af7d-4904-89cb-8c53cd8bd5b8" (UID: "1617a80d-af7d-4904-89cb-8c53cd8bd5b8"). InnerVolumeSpecName "kube-api-access-smzqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:05.767245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:05.767161 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:05.767245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:05.767191 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smzqz\" (UniqueName: \"kubernetes.io/projected/1617a80d-af7d-4904-89cb-8c53cd8bd5b8-kube-api-access-smzqz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:06.442732 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.442704 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" Oct 30 18:11:06.442953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.442707 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8" event={"ID":"1617a80d-af7d-4904-89cb-8c53cd8bd5b8","Type":"ContainerDied","Data":"44b3aa08a4e4400591d6b82283a73bbac10005b9cebaa77cc35c8d56c5bfdd5c"} Oct 30 18:11:06.442953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.442825 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44b3aa08a4e4400591d6b82283a73bbac10005b9cebaa77cc35c8d56c5bfdd5c" Oct 30 18:11:06.856175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.856141 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8"] Oct 30 18:11:06.857142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.857111 4883 cpu_manager.go:414] "containerMap: removing container" podUID="1617a80d-af7d-4904-89cb-8c53cd8bd5b8" containerName="revoke-keys" Oct 30 18:11:06.857142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.857141 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1617a80d-af7d-4904-89cb-8c53cd8bd5b8" containerName="revoke-keys" Oct 30 18:11:06.857346 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.857177 4883 cpu_manager.go:414] "containerMap: removing container" podUID="8632419b-f939-49f2-854a-a1b3903d8560" containerName="revoke-keys" Oct 30 18:11:06.857346 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.857188 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8632419b-f939-49f2-854a-a1b3903d8560" containerName="revoke-keys" Oct 30 18:11:06.857454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.857365 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="1617a80d-af7d-4904-89cb-8c53cd8bd5b8" containerName="revoke-keys" Oct 30 18:11:06.857454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.857387 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="8632419b-f939-49f2-854a-a1b3903d8560" containerName="revoke-keys" Oct 30 18:11:06.864419 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.862332 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:06.881376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.881345 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8"] Oct 30 18:11:06.889216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.889181 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-419d32\"/\"distinct-w4-419d32-kserve-self-signed-certs\"" Oct 30 18:11:06.985325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.985274 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-tmp-dir\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:06.985514 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.985340 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-model-cache\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:06.985514 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.985461 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39fe754a-78b0-4d4e-8275-7ea2dd247508-tls-certs\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:06.985588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.985529 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-dshm\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:06.985588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.985579 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-home\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:06.985658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:06.985595 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbz49\" (UniqueName: \"kubernetes.io/projected/39fe754a-78b0-4d4e-8275-7ea2dd247508-kube-api-access-qbz49\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.094959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.089794 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-tmp-dir\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.094959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.089861 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-model-cache\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.094959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.089923 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39fe754a-78b0-4d4e-8275-7ea2dd247508-tls-certs\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.094959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.090007 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-dshm\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.094959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.090066 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-home\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.094959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.090092 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbz49\" (UniqueName: \"kubernetes.io/projected/39fe754a-78b0-4d4e-8275-7ea2dd247508-kube-api-access-qbz49\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.094959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.091066 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-tmp-dir\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.094959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.091319 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-model-cache\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.094959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.092232 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-home\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.106007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.102131 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39fe754a-78b0-4d4e-8275-7ea2dd247508-tls-certs\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.117909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.115337 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-dshm\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.121978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.120608 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbz49\" (UniqueName: \"kubernetes.io/projected/39fe754a-78b0-4d4e-8275-7ea2dd247508-kube-api-access-qbz49\") pod \"distinct-w4-419d32-kserve-5768cb7fd-d48j8\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.180431 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.180105 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:07.533723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:07.533638 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8"] Oct 30 18:11:08.458977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:08.458921 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" event={"ID":"39fe754a-78b0-4d4e-8275-7ea2dd247508","Type":"ContainerStarted","Data":"c5bba1252b3e325682332247360375920c2575d8e9136843ec4d3e9346f4f51b"} Oct 30 18:11:08.459393 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:08.458988 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" event={"ID":"39fe754a-78b0-4d4e-8275-7ea2dd247508","Type":"ContainerStarted","Data":"1992f9ef2cdf6fd34ac4e3d0770f6717b8a66c70362e0e2080df2e7cd4ec4cdc"} Oct 30 18:11:08.505348 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:08.505282 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podStartSLOduration=2.505265931 podStartE2EDuration="2.505265931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:08.502065278 +0000 UTC m=+1127.459069114" watchObservedRunningTime="2026-10-30 18:11:08.505265931 +0000 UTC m=+1127.462269766" Oct 30 18:11:09.180579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:09.180540 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:14.031070 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:14.031032 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q"] Oct 30 18:11:14.045491 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:14.045442 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f7174b-69c7q"] Oct 30 18:11:15.178704 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:15.178664 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78c4ed74-4e02-41d8-84a2-bb662616dfe4" path="/var/lib/kubelet/pods/78c4ed74-4e02-41d8-84a2-bb662616dfe4/volumes" Oct 30 18:11:17.180612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:17.180581 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:17.193767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:17.193731 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:17.542878 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:17.542344 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:20.034951 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:20.034912 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc"] Oct 30 18:11:20.049067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:20.049032 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958c424a-xzxnc"] Oct 30 18:11:21.182585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.182544 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e2f012-f684-480d-8f06-d91b6ae6020d" path="/var/lib/kubelet/pods/48e2f012-f684-480d-8f06-d91b6ae6020d/volumes" Oct 30 18:11:21.587478 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.587396 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh"] Oct 30 18:11:21.593158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.593127 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:21.614763 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.614732 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh"] Oct 30 18:11:21.620592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.620549 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:11:21.620592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.620563 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:11:21.620842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.620601 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:11:21.620842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.620644 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:11:21.714233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.714203 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30372d63-l5hwh\" (UID: \"f010b93c-e961-4499-a3bc-efbf9588d883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:21.714437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.714272 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30372d63-l5hwh\" (UID: \"f010b93c-e961-4499-a3bc-efbf9588d883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:21.714524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.714506 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxdkk\" (UniqueName: \"kubernetes.io/projected/f010b93c-e961-4499-a3bc-efbf9588d883-kube-api-access-cxdkk\") pod \"maas-api-revoke-sub-30372d63-l5hwh\" (UID: \"f010b93c-e961-4499-a3bc-efbf9588d883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:21.816074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.816039 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxdkk\" (UniqueName: \"kubernetes.io/projected/f010b93c-e961-4499-a3bc-efbf9588d883-kube-api-access-cxdkk\") pod \"maas-api-revoke-sub-30372d63-l5hwh\" (UID: \"f010b93c-e961-4499-a3bc-efbf9588d883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:21.816284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.816120 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30372d63-l5hwh\" (UID: \"f010b93c-e961-4499-a3bc-efbf9588d883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:21.816284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.816171 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30372d63-l5hwh\" (UID: \"f010b93c-e961-4499-a3bc-efbf9588d883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:21.816852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.816826 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30372d63-l5hwh\" (UID: \"f010b93c-e961-4499-a3bc-efbf9588d883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:21.818655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.818635 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30372d63-l5hwh\" (UID: \"f010b93c-e961-4499-a3bc-efbf9588d883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:21.840981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.840868 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxdkk\" (UniqueName: \"kubernetes.io/projected/f010b93c-e961-4499-a3bc-efbf9588d883-kube-api-access-cxdkk\") pod \"maas-api-revoke-sub-30372d63-l5hwh\" (UID: \"f010b93c-e961-4499-a3bc-efbf9588d883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:21.916404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:21.916366 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:22.229477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:22.228589 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh"] Oct 30 18:11:22.229477 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:11:22.229427 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf010b93c_e961_4499_a3bc_efbf9588d883.slice/crio-ab2ddc9de067651860f0681acbd4c59f5c4b2ef052fd95071b3b04574174b93c WatchSource:0}: Error finding container ab2ddc9de067651860f0681acbd4c59f5c4b2ef052fd95071b3b04574174b93c: Status 404 returned error can't find the container with id ab2ddc9de067651860f0681acbd4c59f5c4b2ef052fd95071b3b04574174b93c Oct 30 18:11:22.574540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:22.574484 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" event={"ID":"f010b93c-e961-4499-a3bc-efbf9588d883","Type":"ContainerStarted","Data":"ab2ddc9de067651860f0681acbd4c59f5c4b2ef052fd95071b3b04574174b93c"} Oct 30 18:11:23.589622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:23.589523 4883 generic.go:356] "Generic (PLEG): container finished" podID="f010b93c-e961-4499-a3bc-efbf9588d883" containerID="a488f419e3fb5fdf9bee333a4f36506b29690182eb763048d37b2d2f107a4668" exitCode=0 Oct 30 18:11:23.590248 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:23.589923 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" event={"ID":"f010b93c-e961-4499-a3bc-efbf9588d883","Type":"ContainerDied","Data":"a488f419e3fb5fdf9bee333a4f36506b29690182eb763048d37b2d2f107a4668"} Oct 30 18:11:25.568018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:25.555745 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:11:25.568018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:25.557002 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:25.568018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:25.557272 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:11:25.786828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:25.786782 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:11:25.856161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:25.855788 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:11:25.856161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:25.855988 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:11:25.870789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:25.821129 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:11:26.022923 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.022866 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.051398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.042070 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.139922 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.139812 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.204246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.204205 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8"] Oct 30 18:11:26.235976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.232340 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.293976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.275524 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:26.293976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.275594 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.371030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.370998 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:26.392220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.392124 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8"] Oct 30 18:11:26.416153 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.393636 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:26.606961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.603336 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.635310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.635264 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-openshift-ca\") pod \"f010b93c-e961-4499-a3bc-efbf9588d883\" (UID: \"f010b93c-e961-4499-a3bc-efbf9588d883\") " Oct 30 18:11:26.662705 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.662607 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-xks-ca\") pod \"f010b93c-e961-4499-a3bc-efbf9588d883\" (UID: \"f010b93c-e961-4499-a3bc-efbf9588d883\") " Oct 30 18:11:26.663071 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.663045 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f010b93c-e961-4499-a3bc-efbf9588d883-kube-api-access-cxdkk\" (UniqueName: \"kubernetes.io/projected/f010b93c-e961-4499-a3bc-efbf9588d883-kube-api-access-cxdkk\") pod \"f010b93c-e961-4499-a3bc-efbf9588d883\" (UID: \"f010b93c-e961-4499-a3bc-efbf9588d883\") " Oct 30 18:11:26.670409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.670374 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljnd\" (UniqueName: \"kubernetes.io/projected/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-kube-api-access-sljnd\") pod \"maas-api-revoke-sub-48677ed6-vgtp8\" (UID: \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:26.671096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.670711 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48677ed6-vgtp8\" (UID: \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:26.671808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.671772 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48677ed6-vgtp8\" (UID: \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:26.673110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.642269 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:11:26.678031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.645529 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-openshift-ca" pod "f010b93c-e961-4499-a3bc-efbf9588d883" (UID: "f010b93c-e961-4499-a3bc-efbf9588d883"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:26.703182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.641102 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:26.705119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.643915 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:27.084605 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.063117 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:11:27.084605 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.063306 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:11:27.084605 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.827063 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.134366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.881192 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.134366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.882123 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.134366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.882155 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.134366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.124368 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:11:27.134366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.124486 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:11:27.134366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.882233 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.134366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.031368 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:11:27.134366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.045142 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.134366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.062869 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48677ed6-vgtp8\" (UID: \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:27.134366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.131629 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:11:27.134366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.131717 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:11:27.134366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.131809 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:11:27.155989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.863089 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:11:27.166089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:26.862480 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48677ed6-vgtp8\" (UID: \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:27.170970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.168137 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljnd\" (UniqueName: \"kubernetes.io/projected/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-kube-api-access-sljnd\") pod \"maas-api-revoke-sub-48677ed6-vgtp8\" (UID: \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:27.178975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.171279 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48677ed6-vgtp8\" (UID: \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:27.321996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.179420 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:11:27.363321 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.361217 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" event={"ID":"f010b93c-e961-4499-a3bc-efbf9588d883","Type":"ContainerDied","Data":"ab2ddc9de067651860f0681acbd4c59f5c4b2ef052fd95071b3b04574174b93c"} Oct 30 18:11:27.363321 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.361292 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab2ddc9de067651860f0681acbd4c59f5c4b2ef052fd95071b3b04574174b93c" Oct 30 18:11:27.366969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.259362 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.367205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.326561 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48677ed6-vgtp8\" (UID: \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:27.367351 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.354666 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:27.367484 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.259425 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:11:27.402961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.382348 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljnd\" (UniqueName: \"kubernetes.io/projected/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-kube-api-access-sljnd\") pod \"maas-api-revoke-sub-48677ed6-vgtp8\" (UID: \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:27.402961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.278610 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh" Oct 30 18:11:27.402961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.401580 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-xks-ca" pod "f010b93c-e961-4499-a3bc-efbf9588d883" (UID: "f010b93c-e961-4499-a3bc-efbf9588d883"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:27.604961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.568019 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f010b93c-e961-4499-a3bc-efbf9588d883-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:27.613762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.608860 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.670840 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.669715 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:27.696781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.674602 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:27.726016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:27.705106 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.150210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.146713 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.150210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.146889 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:11:28.150210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.147037 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.150210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.147139 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:11:28.150210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.147225 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:11:28.198960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.183216 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f010b93c-e961-4499-a3bc-efbf9588d883-kube-api-access-cxdkk" pod "f010b93c-e961-4499-a3bc-efbf9588d883" (UID: "f010b93c-e961-4499-a3bc-efbf9588d883"). InnerVolumeSpecName "kube-api-access-cxdkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:28.198960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.183321 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.556961 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:11:28.556493 4883 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.195s" Oct 30 18:11:28.556961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.556588 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:11:28.556961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.556678 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:11:28.615962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.578038 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxdkk\" (UniqueName: \"kubernetes.io/projected/f010b93c-e961-4499-a3bc-efbf9588d883-kube-api-access-cxdkk\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:28.615962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.592664 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:28.615962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.592714 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.615962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.593688 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Oct 30 18:11:28.615962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.594192 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.615962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.594278 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:11:28.615962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.594592 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:11:28.671171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.671112 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:28.676191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.676153 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:11:28.721576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.714201 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:11:28.966960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:28.966133 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b"] Oct 30 18:11:29.178958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.173226 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.178958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.173630 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:11:29.178958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.173893 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:11:29.178958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.174153 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:11:29.178958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.174385 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:11:29.273098 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.271335 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 30 18:11:29.315532 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.276507 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.462972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.438058 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:11:29.500467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.500350 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.525325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.523508 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.597211 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.597138 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.598916 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.598850 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.602927 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.602869 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:11:29.758662 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.681782 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.758662 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.681825 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:29.772899 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:29.764443 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:11:30.138810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.090511 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:11:30.241075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.118481 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.241336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.215036 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.251968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.215079 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:11:30.277963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.215105 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.280676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.280646 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16135d4b-npj7b"] Oct 30 18:11:30.281020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.280984 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.462328 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.462221 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.536272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.536224 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.575400 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.550117 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.620031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.617354 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.701294 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.701262 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:11:30.715055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.714959 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" event={"ID":"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8","Type":"ContainerStarted","Data":"3f09db73b3351edb84b5be62f4fa0c208812bb3f13cd31cd2aa24b6737329fb3"} Oct 30 18:11:30.735966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.735353 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:11:30.781225 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.763033 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.781225 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.763351 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.781798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.781281 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:30.781798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.781359 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.790055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.786113 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:30.843961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:30.824073 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.296096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:31.296041 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:11:31.302735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:31.302689 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.313191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:31.303382 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.372719 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:31.372675 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e72e77-96d1-44bd-9085-b688edd838dc" path="/var/lib/kubelet/pods/60e72e77-96d1-44bd-9085-b688edd838dc/volumes" Oct 30 18:11:31.597025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:31.596276 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:11:31.637272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:31.634664 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:11:31.713960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:31.691325 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.753958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:31.740026 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.851157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:31.842113 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.851157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:31.842518 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:31.851157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:31.847476 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:11:31.851157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:31.847825 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:11:32.038323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.038295 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8"] Oct 30 18:11:32.039222 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.039198 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:11:32.039393 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.039384 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:11:32.050977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.043013 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:11:32.050977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.046921 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:11:32.072186 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.072139 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" event={"ID":"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8","Type":"ContainerStarted","Data":"fe1502a336923aee3155b2bd2965b3acf68e5c3ca89185603ef782b7cdd72f7d"} Oct 30 18:11:32.080427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.080400 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:11:32.133562 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.133480 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:11:32.190454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.189599 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:11:32.257967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.257068 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:11:32.257967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.257135 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:11:32.263072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.260234 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:11:32.318189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:32.318129 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" podStartSLOduration=7.318107953 podStartE2EDuration="7.318107953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:32.248531161 +0000 UTC m=+1151.205534998" watchObservedRunningTime="2026-10-30 18:11:32.318107953 +0000 UTC m=+1151.275111789" Oct 30 18:11:33.080563 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:33.080525 4883 generic.go:356] "Generic (PLEG): container finished" podID="86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8" containerID="fe1502a336923aee3155b2bd2965b3acf68e5c3ca89185603ef782b7cdd72f7d" exitCode=0 Oct 30 18:11:33.081068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:33.080617 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" event={"ID":"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8","Type":"ContainerDied","Data":"fe1502a336923aee3155b2bd2965b3acf68e5c3ca89185603ef782b7cdd72f7d"} Oct 30 18:11:34.282842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:34.282810 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:34.416953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:34.416904 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-kube-api-access-sljnd\" (UniqueName: \"kubernetes.io/projected/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-kube-api-access-sljnd\") pod \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\" (UID: \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\") " Oct 30 18:11:34.417163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:34.417064 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-openshift-ca\") pod \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\" (UID: \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\") " Oct 30 18:11:34.417265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:34.417248 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-xks-ca\") pod \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\" (UID: \"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8\") " Oct 30 18:11:34.417443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:34.417409 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-openshift-ca" pod "86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8" (UID: "86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:34.417767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:34.417749 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:34.419274 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:34.419248 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-xks-ca" pod "86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8" (UID: "86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:34.419417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:34.419399 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-kube-api-access-sljnd" pod "86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8" (UID: "86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8"). InnerVolumeSpecName "kube-api-access-sljnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:34.519323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:34.519282 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:34.519323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:34.519315 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljnd\" (UniqueName: \"kubernetes.io/projected/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8-kube-api-access-sljnd\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:35.035753 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:35.035695 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr"] Oct 30 18:11:35.048840 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:35.048804 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bf06ca-xlnjr"] Oct 30 18:11:35.084314 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:11:35.084267 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:11:35.101203 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:35.101161 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" event={"ID":"86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8","Type":"ContainerDied","Data":"3f09db73b3351edb84b5be62f4fa0c208812bb3f13cd31cd2aa24b6737329fb3"} Oct 30 18:11:35.101203 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:35.101197 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8" Oct 30 18:11:35.101418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:35.101266 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f09db73b3351edb84b5be62f4fa0c208812bb3f13cd31cd2aa24b6737329fb3" Oct 30 18:11:35.180463 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:35.180430 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb444d7-c764-4f6c-bdbf-aeca289c89c0" path="/var/lib/kubelet/pods/ddb444d7-c764-4f6c-bdbf-aeca289c89c0/volumes" Oct 30 18:11:39.907335 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.907295 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh"] Oct 30 18:11:39.907868 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.907855 4883 cpu_manager.go:414] "containerMap: removing container" podUID="f010b93c-e961-4499-a3bc-efbf9588d883" containerName="revoke-keys" Oct 30 18:11:39.907868 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.907867 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f010b93c-e961-4499-a3bc-efbf9588d883" containerName="revoke-keys" Oct 30 18:11:39.908039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.907877 4883 cpu_manager.go:414] "containerMap: removing container" podUID="86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8" containerName="revoke-keys" Oct 30 18:11:39.908039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.907883 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8" containerName="revoke-keys" Oct 30 18:11:39.908039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.908019 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="f010b93c-e961-4499-a3bc-efbf9588d883" containerName="revoke-keys" Oct 30 18:11:39.908039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.908028 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8" containerName="revoke-keys" Oct 30 18:11:39.911578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.911552 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:39.927836 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.927774 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:11:39.932592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.932558 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh"] Oct 30 18:11:39.942982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.940716 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:11:39.942982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.940797 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:11:39.942982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.940993 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:11:39.989030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.988992 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mb7j\" (UniqueName: \"kubernetes.io/projected/ec110c06-af6a-40a4-9423-9f8b661fa70d-kube-api-access-5mb7j\") pod \"maas-api-revoke-sub-546c5d31-kv6rh\" (UID: \"ec110c06-af6a-40a4-9423-9f8b661fa70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:39.989030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.989038 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-546c5d31-kv6rh\" (UID: \"ec110c06-af6a-40a4-9423-9f8b661fa70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:39.989253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:39.989059 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-546c5d31-kv6rh\" (UID: \"ec110c06-af6a-40a4-9423-9f8b661fa70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:40.091203 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.091167 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mb7j\" (UniqueName: \"kubernetes.io/projected/ec110c06-af6a-40a4-9423-9f8b661fa70d-kube-api-access-5mb7j\") pod \"maas-api-revoke-sub-546c5d31-kv6rh\" (UID: \"ec110c06-af6a-40a4-9423-9f8b661fa70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:40.091422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.091218 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-546c5d31-kv6rh\" (UID: \"ec110c06-af6a-40a4-9423-9f8b661fa70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:40.091422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.091336 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-546c5d31-kv6rh\" (UID: \"ec110c06-af6a-40a4-9423-9f8b661fa70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:40.092027 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.092007 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-546c5d31-kv6rh\" (UID: \"ec110c06-af6a-40a4-9423-9f8b661fa70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:40.093815 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.093790 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-546c5d31-kv6rh\" (UID: \"ec110c06-af6a-40a4-9423-9f8b661fa70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:40.115915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.115874 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mb7j\" (UniqueName: \"kubernetes.io/projected/ec110c06-af6a-40a4-9423-9f8b661fa70d-kube-api-access-5mb7j\") pod \"maas-api-revoke-sub-546c5d31-kv6rh\" (UID: \"ec110c06-af6a-40a4-9423-9f8b661fa70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:40.225536 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.225400 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:40.538611 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.538579 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh"] Oct 30 18:11:40.539519 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:11:40.539495 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec110c06_af6a_40a4_9423_9f8b661fa70d.slice/crio-50bdda8b98cb6d989da39253d60ae2719fb2fe7d8db596e2821576e35fcb85c4 WatchSource:0}: Error finding container 50bdda8b98cb6d989da39253d60ae2719fb2fe7d8db596e2821576e35fcb85c4: Status 404 returned error can't find the container with id 50bdda8b98cb6d989da39253d60ae2719fb2fe7d8db596e2821576e35fcb85c4 Oct 30 18:11:40.796283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.796200 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th"] Oct 30 18:11:40.801832 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.801806 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:40.821984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.821918 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th"] Oct 30 18:11:40.903345 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.903308 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsq2v\" (UniqueName: \"kubernetes.io/projected/fa42df06-1e0b-4380-a785-0640e61cf872-kube-api-access-tsq2v\") pod \"maas-api-revoke-sub-2f12911a-kh9th\" (UID: \"fa42df06-1e0b-4380-a785-0640e61cf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:40.903571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.903366 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f12911a-kh9th\" (UID: \"fa42df06-1e0b-4380-a785-0640e61cf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:40.903571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:40.903485 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f12911a-kh9th\" (UID: \"fa42df06-1e0b-4380-a785-0640e61cf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:41.005648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:41.005596 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsq2v\" (UniqueName: \"kubernetes.io/projected/fa42df06-1e0b-4380-a785-0640e61cf872-kube-api-access-tsq2v\") pod \"maas-api-revoke-sub-2f12911a-kh9th\" (UID: \"fa42df06-1e0b-4380-a785-0640e61cf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:41.006095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:41.005662 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f12911a-kh9th\" (UID: \"fa42df06-1e0b-4380-a785-0640e61cf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:41.006095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:41.005732 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f12911a-kh9th\" (UID: \"fa42df06-1e0b-4380-a785-0640e61cf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:41.006466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:41.006448 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f12911a-kh9th\" (UID: \"fa42df06-1e0b-4380-a785-0640e61cf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:41.008194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:41.008170 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f12911a-kh9th\" (UID: \"fa42df06-1e0b-4380-a785-0640e61cf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:41.030352 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:41.030319 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsq2v\" (UniqueName: \"kubernetes.io/projected/fa42df06-1e0b-4380-a785-0640e61cf872-kube-api-access-tsq2v\") pod \"maas-api-revoke-sub-2f12911a-kh9th\" (UID: \"fa42df06-1e0b-4380-a785-0640e61cf872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:41.116974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:41.116922 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:41.148494 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:41.148457 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" event={"ID":"ec110c06-af6a-40a4-9423-9f8b661fa70d","Type":"ContainerStarted","Data":"c498901dc9fdee3892d24efdd6cba07c8bb13dfdf9f99f0b2bd9da5924a25b8e"} Oct 30 18:11:41.148703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:41.148513 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" event={"ID":"ec110c06-af6a-40a4-9423-9f8b661fa70d","Type":"ContainerStarted","Data":"50bdda8b98cb6d989da39253d60ae2719fb2fe7d8db596e2821576e35fcb85c4"} Oct 30 18:11:41.197910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:41.197845 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" podStartSLOduration=2.19782514 podStartE2EDuration="2.19782514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:41.197264302 +0000 UTC m=+1160.154268139" watchObservedRunningTime="2026-10-30 18:11:41.19782514 +0000 UTC m=+1160.154828977" Oct 30 18:11:41.438635 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:41.438599 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th"] Oct 30 18:11:41.439459 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:11:41.439431 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa42df06_1e0b_4380_a785_0640e61cf872.slice/crio-489575a31df9ce4972708d1c801c214564e4df4adcecde1e21890af862b6d18d WatchSource:0}: Error finding container 489575a31df9ce4972708d1c801c214564e4df4adcecde1e21890af862b6d18d: Status 404 returned error can't find the container with id 489575a31df9ce4972708d1c801c214564e4df4adcecde1e21890af862b6d18d Oct 30 18:11:42.158281 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:42.158243 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" event={"ID":"fa42df06-1e0b-4380-a785-0640e61cf872","Type":"ContainerStarted","Data":"94b8f0abd3b7c7b3a4920545e65becf6d6452535140b61b3d662a6de12d17d4a"} Oct 30 18:11:42.158764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:42.158301 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" event={"ID":"fa42df06-1e0b-4380-a785-0640e61cf872","Type":"ContainerStarted","Data":"489575a31df9ce4972708d1c801c214564e4df4adcecde1e21890af862b6d18d"} Oct 30 18:11:42.161522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:42.161488 4883 generic.go:356] "Generic (PLEG): container finished" podID="ec110c06-af6a-40a4-9423-9f8b661fa70d" containerID="c498901dc9fdee3892d24efdd6cba07c8bb13dfdf9f99f0b2bd9da5924a25b8e" exitCode=0 Oct 30 18:11:42.161710 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:42.161608 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" event={"ID":"ec110c06-af6a-40a4-9423-9f8b661fa70d","Type":"ContainerDied","Data":"c498901dc9fdee3892d24efdd6cba07c8bb13dfdf9f99f0b2bd9da5924a25b8e"} Oct 30 18:11:42.207982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:42.207912 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" podStartSLOduration=2.207897674 podStartE2EDuration="2.207897674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:42.204601886 +0000 UTC m=+1161.161605724" watchObservedRunningTime="2026-10-30 18:11:42.207897674 +0000 UTC m=+1161.164901510" Oct 30 18:11:43.177059 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:43.177023 4883 generic.go:356] "Generic (PLEG): container finished" podID="fa42df06-1e0b-4380-a785-0640e61cf872" containerID="94b8f0abd3b7c7b3a4920545e65becf6d6452535140b61b3d662a6de12d17d4a" exitCode=0 Oct 30 18:11:43.184337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:43.184291 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" event={"ID":"fa42df06-1e0b-4380-a785-0640e61cf872","Type":"ContainerDied","Data":"94b8f0abd3b7c7b3a4920545e65becf6d6452535140b61b3d662a6de12d17d4a"} Oct 30 18:11:43.362007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:43.361980 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:43.448708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:43.448614 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec110c06-af6a-40a4-9423-9f8b661fa70d-kube-api-access-5mb7j\" (UniqueName: \"kubernetes.io/projected/ec110c06-af6a-40a4-9423-9f8b661fa70d-kube-api-access-5mb7j\") pod \"ec110c06-af6a-40a4-9423-9f8b661fa70d\" (UID: \"ec110c06-af6a-40a4-9423-9f8b661fa70d\") " Oct 30 18:11:43.448872 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:43.448843 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-xks-ca\") pod \"ec110c06-af6a-40a4-9423-9f8b661fa70d\" (UID: \"ec110c06-af6a-40a4-9423-9f8b661fa70d\") " Oct 30 18:11:43.449000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:43.448895 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-openshift-ca\") pod \"ec110c06-af6a-40a4-9423-9f8b661fa70d\" (UID: \"ec110c06-af6a-40a4-9423-9f8b661fa70d\") " Oct 30 18:11:43.449654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:43.449629 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-openshift-ca" pod "ec110c06-af6a-40a4-9423-9f8b661fa70d" (UID: "ec110c06-af6a-40a4-9423-9f8b661fa70d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:43.451581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:43.451545 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-xks-ca" pod "ec110c06-af6a-40a4-9423-9f8b661fa70d" (UID: "ec110c06-af6a-40a4-9423-9f8b661fa70d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:43.451820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:43.451789 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec110c06-af6a-40a4-9423-9f8b661fa70d-kube-api-access-5mb7j" pod "ec110c06-af6a-40a4-9423-9f8b661fa70d" (UID: "ec110c06-af6a-40a4-9423-9f8b661fa70d"). InnerVolumeSpecName "kube-api-access-5mb7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:43.551064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:43.551026 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:43.551064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:43.551059 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mb7j\" (UniqueName: \"kubernetes.io/projected/ec110c06-af6a-40a4-9423-9f8b661fa70d-kube-api-access-5mb7j\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:43.551064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:43.551069 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec110c06-af6a-40a4-9423-9f8b661fa70d-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:44.193955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.192038 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" event={"ID":"ec110c06-af6a-40a4-9423-9f8b661fa70d","Type":"ContainerDied","Data":"50bdda8b98cb6d989da39253d60ae2719fb2fe7d8db596e2821576e35fcb85c4"} Oct 30 18:11:44.193955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.192116 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50bdda8b98cb6d989da39253d60ae2719fb2fe7d8db596e2821576e35fcb85c4" Oct 30 18:11:44.193955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.192211 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh" Oct 30 18:11:44.389509 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.389479 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:44.464205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.464087 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa42df06-1e0b-4380-a785-0640e61cf872-kube-api-access-tsq2v\" (UniqueName: \"kubernetes.io/projected/fa42df06-1e0b-4380-a785-0640e61cf872-kube-api-access-tsq2v\") pod \"fa42df06-1e0b-4380-a785-0640e61cf872\" (UID: \"fa42df06-1e0b-4380-a785-0640e61cf872\") " Oct 30 18:11:44.464205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.464184 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-xks-ca\") pod \"fa42df06-1e0b-4380-a785-0640e61cf872\" (UID: \"fa42df06-1e0b-4380-a785-0640e61cf872\") " Oct 30 18:11:44.464450 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.464434 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-openshift-ca\") pod \"fa42df06-1e0b-4380-a785-0640e61cf872\" (UID: \"fa42df06-1e0b-4380-a785-0640e61cf872\") " Oct 30 18:11:44.464927 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.464896 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-openshift-ca" pod "fa42df06-1e0b-4380-a785-0640e61cf872" (UID: "fa42df06-1e0b-4380-a785-0640e61cf872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:44.465344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.465319 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:44.467291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.467258 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-xks-ca" pod "fa42df06-1e0b-4380-a785-0640e61cf872" (UID: "fa42df06-1e0b-4380-a785-0640e61cf872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:44.467414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.467283 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa42df06-1e0b-4380-a785-0640e61cf872-kube-api-access-tsq2v" pod "fa42df06-1e0b-4380-a785-0640e61cf872" (UID: "fa42df06-1e0b-4380-a785-0640e61cf872"). InnerVolumeSpecName "kube-api-access-tsq2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:44.566184 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.566148 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsq2v\" (UniqueName: \"kubernetes.io/projected/fa42df06-1e0b-4380-a785-0640e61cf872-kube-api-access-tsq2v\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:44.566184 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:44.566181 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa42df06-1e0b-4380-a785-0640e61cf872-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:45.036033 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:45.035985 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6"] Oct 30 18:11:45.044097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:45.044059 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59b11feb-gvwn6"] Oct 30 18:11:45.180651 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:45.180610 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea47f43-d2d2-47c6-a183-99297339b3f4" path="/var/lib/kubelet/pods/8ea47f43-d2d2-47c6-a183-99297339b3f4/volumes" Oct 30 18:11:45.201081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:45.201052 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" Oct 30 18:11:45.201081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:45.201068 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th" event={"ID":"fa42df06-1e0b-4380-a785-0640e61cf872","Type":"ContainerDied","Data":"489575a31df9ce4972708d1c801c214564e4df4adcecde1e21890af862b6d18d"} Oct 30 18:11:45.201502 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:45.201119 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="489575a31df9ce4972708d1c801c214564e4df4adcecde1e21890af862b6d18d" Oct 30 18:11:48.405058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.405019 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr"] Oct 30 18:11:48.405837 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.405817 4883 cpu_manager.go:414] "containerMap: removing container" podUID="fa42df06-1e0b-4380-a785-0640e61cf872" containerName="revoke-keys" Oct 30 18:11:48.405988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.405840 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa42df06-1e0b-4380-a785-0640e61cf872" containerName="revoke-keys" Oct 30 18:11:48.405988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.405858 4883 cpu_manager.go:414] "containerMap: removing container" podUID="ec110c06-af6a-40a4-9423-9f8b661fa70d" containerName="revoke-keys" Oct 30 18:11:48.405988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.405866 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec110c06-af6a-40a4-9423-9f8b661fa70d" containerName="revoke-keys" Oct 30 18:11:48.406166 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.406077 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa42df06-1e0b-4380-a785-0640e61cf872" containerName="revoke-keys" Oct 30 18:11:48.406166 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.406092 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec110c06-af6a-40a4-9423-9f8b661fa70d" containerName="revoke-keys" Oct 30 18:11:48.410246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.410216 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.439354 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.439306 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr"] Oct 30 18:11:48.439755 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.439736 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-419d32\"/\"distinct-2-w4-419d32-kserve-self-signed-certs\"" Oct 30 18:11:48.522577 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.522533 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-model-cache\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.522783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.522605 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tmp-dir\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.522783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.522719 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdp2z\" (UniqueName: \"kubernetes.io/projected/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-kube-api-access-sdp2z\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.522783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.522769 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-home\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.522911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.522862 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-dshm\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.523031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.523017 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tls-certs\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.626958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.626870 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tls-certs\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.627179 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.627019 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-model-cache\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.627179 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.627094 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tmp-dir\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.627179 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.627155 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdp2z\" (UniqueName: \"kubernetes.io/projected/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-kube-api-access-sdp2z\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.627349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.627191 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-home\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.627349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.627257 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-dshm\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.632992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.627988 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tmp-dir\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.632992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.628923 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-home\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.632992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.629181 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-model-cache\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.650962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.646309 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-dshm\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.668656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.663840 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdp2z\" (UniqueName: \"kubernetes.io/projected/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-kube-api-access-sdp2z\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.673123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.671897 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tls-certs\") pod \"distinct-2-w4-419d32-kserve-865477b849-tzvwr\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:48.745014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:48.742272 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:49.093521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:49.093492 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr"] Oct 30 18:11:49.095385 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:11:49.095284 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f2e9af9_7137_4b2e_a0ea_beea94c4cbed.slice/crio-e0c923d3571e98bfcf8ec6c9dc488a200b10cd52c5514e127ff9e899c3eda013 WatchSource:0}: Error finding container e0c923d3571e98bfcf8ec6c9dc488a200b10cd52c5514e127ff9e899c3eda013: Status 404 returned error can't find the container with id e0c923d3571e98bfcf8ec6c9dc488a200b10cd52c5514e127ff9e899c3eda013 Oct 30 18:11:49.238691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:49.238652 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" event={"ID":"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed","Type":"ContainerStarted","Data":"4a939b0c861da5c50250ffb4e1e2fb0357edc2289451b10c531b06542dcc6579"} Oct 30 18:11:49.238691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:49.238694 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" event={"ID":"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed","Type":"ContainerStarted","Data":"e0c923d3571e98bfcf8ec6c9dc488a200b10cd52c5514e127ff9e899c3eda013"} Oct 30 18:11:49.283853 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:49.283792 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podStartSLOduration=1.283778063 podStartE2EDuration="1.283778063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:49.282253023 +0000 UTC m=+1168.239256860" watchObservedRunningTime="2026-10-30 18:11:49.283778063 +0000 UTC m=+1168.240781898" Oct 30 18:11:49.743120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:49.743082 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:50.242180 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.242144 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x"] Oct 30 18:11:50.250482 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.250446 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:50.267575 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.267540 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x"] Oct 30 18:11:50.274546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.274496 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:11:50.274734 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.274570 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:11:50.274734 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.274637 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:11:50.274734 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.274720 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:11:50.358636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.358594 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfwx9\" (UniqueName: \"kubernetes.io/projected/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-kube-api-access-wfwx9\") pod \"maas-api-revoke-sub-a3173e1b-fbm5x\" (UID: \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:50.358636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.358636 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3173e1b-fbm5x\" (UID: \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:50.358850 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.358768 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3173e1b-fbm5x\" (UID: \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:50.460215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.460169 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfwx9\" (UniqueName: \"kubernetes.io/projected/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-kube-api-access-wfwx9\") pod \"maas-api-revoke-sub-a3173e1b-fbm5x\" (UID: \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:50.460215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.460216 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3173e1b-fbm5x\" (UID: \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:50.460477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.460302 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3173e1b-fbm5x\" (UID: \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:50.461054 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.461029 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3173e1b-fbm5x\" (UID: \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:50.462872 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.462849 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3173e1b-fbm5x\" (UID: \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:50.484871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.484839 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfwx9\" (UniqueName: \"kubernetes.io/projected/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-kube-api-access-wfwx9\") pod \"maas-api-revoke-sub-a3173e1b-fbm5x\" (UID: \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:50.569428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.569343 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:50.879997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:50.879944 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x"] Oct 30 18:11:50.881017 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:11:50.880979 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3a8aae_4aa1_44d5_ad5a_9d80035bfbf3.slice/crio-3490489cb239db62e9053e91c3d88ae597d9000cadfcb2769e1582555e555b6b WatchSource:0}: Error finding container 3490489cb239db62e9053e91c3d88ae597d9000cadfcb2769e1582555e555b6b: Status 404 returned error can't find the container with id 3490489cb239db62e9053e91c3d88ae597d9000cadfcb2769e1582555e555b6b Oct 30 18:11:51.260041 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:51.260001 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" event={"ID":"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3","Type":"ContainerStarted","Data":"027f59d0944f700d2e5fe10e8d302a5052d2a0bfe97c11082236952fc06ea75d"} Oct 30 18:11:51.260225 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:51.260089 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" event={"ID":"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3","Type":"ContainerStarted","Data":"3490489cb239db62e9053e91c3d88ae597d9000cadfcb2769e1582555e555b6b"} Oct 30 18:11:51.305083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:51.305026 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" podStartSLOduration=1.305009787 podStartE2EDuration="1.305009787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:51.301849498 +0000 UTC m=+1170.258853335" watchObservedRunningTime="2026-10-30 18:11:51.305009787 +0000 UTC m=+1170.262013622" Oct 30 18:11:52.268577 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:52.268543 4883 generic.go:356] "Generic (PLEG): container finished" podID="3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3" containerID="027f59d0944f700d2e5fe10e8d302a5052d2a0bfe97c11082236952fc06ea75d" exitCode=0 Oct 30 18:11:52.269001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:52.268634 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" event={"ID":"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3","Type":"ContainerDied","Data":"027f59d0944f700d2e5fe10e8d302a5052d2a0bfe97c11082236952fc06ea75d"} Oct 30 18:11:53.457494 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:53.457462 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:53.612986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:53.610837 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-kube-api-access-wfwx9\" (UniqueName: \"kubernetes.io/projected/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-kube-api-access-wfwx9\") pod \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\" (UID: \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\") " Oct 30 18:11:53.612986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:53.611181 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-xks-ca\") pod \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\" (UID: \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\") " Oct 30 18:11:53.612986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:53.611244 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-openshift-ca\") pod \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\" (UID: \"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3\") " Oct 30 18:11:53.636959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:53.625157 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-openshift-ca" pod "3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3" (UID: "3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:53.666183 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:53.654144 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-kube-api-access-wfwx9" pod "3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3" (UID: "3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3"). InnerVolumeSpecName "kube-api-access-wfwx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:53.684965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:53.684291 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-xks-ca" pod "3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3" (UID: "3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:53.725967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:53.715130 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:53.725967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:53.715169 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:53.725967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:53.715186 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfwx9\" (UniqueName: \"kubernetes.io/projected/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3-kube-api-access-wfwx9\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:11:54.291963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:54.290139 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" Oct 30 18:11:54.291963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:54.290151 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x" event={"ID":"3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3","Type":"ContainerDied","Data":"3490489cb239db62e9053e91c3d88ae597d9000cadfcb2769e1582555e555b6b"} Oct 30 18:11:54.291963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:54.290206 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3490489cb239db62e9053e91c3d88ae597d9000cadfcb2769e1582555e555b6b" Oct 30 18:11:55.030548 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:55.030506 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9"] Oct 30 18:11:55.044175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:55.044140 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9f3fa4-pm9q9"] Oct 30 18:11:55.188233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:55.188182 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3941d7-23ca-4d49-b979-15fb991148c7" path="/var/lib/kubelet/pods/0f3941d7-23ca-4d49-b979-15fb991148c7/volumes" Oct 30 18:11:58.743203 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:58.743164 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:58.758351 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:58.758309 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:11:59.345190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:11:59.345155 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:12:05.080839 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:12:05.080800 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:12:13.281044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.281007 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9"] Oct 30 18:12:13.281852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.281831 4883 cpu_manager.go:414] "containerMap: removing container" podUID="3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3" containerName="revoke-keys" Oct 30 18:12:13.281977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.281853 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3" containerName="revoke-keys" Oct 30 18:12:13.282094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.282079 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3" containerName="revoke-keys" Oct 30 18:12:13.285917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.285893 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:13.305733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.305700 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9"] Oct 30 18:12:13.314087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.314056 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:12:13.314087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.314075 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:12:13.314299 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.314053 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:12:13.314299 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.314168 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:12:13.427129 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.427093 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b5160f3-2bmh9\" (UID: \"973674e3-e3f7-4193-8649-2ba7854458b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:13.427328 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.427222 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhjgd\" (UniqueName: \"kubernetes.io/projected/973674e3-e3f7-4193-8649-2ba7854458b8-kube-api-access-zhjgd\") pod \"maas-api-revoke-sub-1b5160f3-2bmh9\" (UID: \"973674e3-e3f7-4193-8649-2ba7854458b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:13.427394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.427368 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b5160f3-2bmh9\" (UID: \"973674e3-e3f7-4193-8649-2ba7854458b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:13.529263 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.529215 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b5160f3-2bmh9\" (UID: \"973674e3-e3f7-4193-8649-2ba7854458b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:13.529444 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.529296 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhjgd\" (UniqueName: \"kubernetes.io/projected/973674e3-e3f7-4193-8649-2ba7854458b8-kube-api-access-zhjgd\") pod \"maas-api-revoke-sub-1b5160f3-2bmh9\" (UID: \"973674e3-e3f7-4193-8649-2ba7854458b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:13.529444 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.529389 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b5160f3-2bmh9\" (UID: \"973674e3-e3f7-4193-8649-2ba7854458b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:13.530093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.530068 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b5160f3-2bmh9\" (UID: \"973674e3-e3f7-4193-8649-2ba7854458b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:13.532009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.531949 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b5160f3-2bmh9\" (UID: \"973674e3-e3f7-4193-8649-2ba7854458b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:13.553917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.553884 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhjgd\" (UniqueName: \"kubernetes.io/projected/973674e3-e3f7-4193-8649-2ba7854458b8-kube-api-access-zhjgd\") pod \"maas-api-revoke-sub-1b5160f3-2bmh9\" (UID: \"973674e3-e3f7-4193-8649-2ba7854458b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:13.600439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.600400 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:13.911951 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:13.911903 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9"] Oct 30 18:12:13.914242 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:12:13.914213 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973674e3_e3f7_4193_8649_2ba7854458b8.slice/crio-41dbd4fdc5bb36bdda21fa0a92276b36089540afba1cf2b93d4d400a2ed01842 WatchSource:0}: Error finding container 41dbd4fdc5bb36bdda21fa0a92276b36089540afba1cf2b93d4d400a2ed01842: Status 404 returned error can't find the container with id 41dbd4fdc5bb36bdda21fa0a92276b36089540afba1cf2b93d4d400a2ed01842 Oct 30 18:12:14.459020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:14.458896 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" event={"ID":"973674e3-e3f7-4193-8649-2ba7854458b8","Type":"ContainerStarted","Data":"548967de468c8408243570da9ba6dd2f8e7d5aae1c90da41ee478f0ccde00f14"} Oct 30 18:12:14.459020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:14.458986 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" event={"ID":"973674e3-e3f7-4193-8649-2ba7854458b8","Type":"ContainerStarted","Data":"41dbd4fdc5bb36bdda21fa0a92276b36089540afba1cf2b93d4d400a2ed01842"} Oct 30 18:12:14.502850 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:14.502792 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" podStartSLOduration=1.5027766310000001 podStartE2EDuration="1.502776631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:14.501129011 +0000 UTC m=+1193.458132849" watchObservedRunningTime="2026-10-30 18:12:14.502776631 +0000 UTC m=+1193.459780467" Oct 30 18:12:15.468986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:15.467927 4883 generic.go:356] "Generic (PLEG): container finished" podID="973674e3-e3f7-4193-8649-2ba7854458b8" containerID="548967de468c8408243570da9ba6dd2f8e7d5aae1c90da41ee478f0ccde00f14" exitCode=0 Oct 30 18:12:15.468986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:15.467964 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" event={"ID":"973674e3-e3f7-4193-8649-2ba7854458b8","Type":"ContainerDied","Data":"548967de468c8408243570da9ba6dd2f8e7d5aae1c90da41ee478f0ccde00f14"} Oct 30 18:12:16.657229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:16.657197 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:16.672831 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:16.672796 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/973674e3-e3f7-4193-8649-2ba7854458b8-kube-api-access-zhjgd\" (UniqueName: \"kubernetes.io/projected/973674e3-e3f7-4193-8649-2ba7854458b8-kube-api-access-zhjgd\") pod \"973674e3-e3f7-4193-8649-2ba7854458b8\" (UID: \"973674e3-e3f7-4193-8649-2ba7854458b8\") " Oct 30 18:12:16.673018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:16.672997 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-xks-ca\") pod \"973674e3-e3f7-4193-8649-2ba7854458b8\" (UID: \"973674e3-e3f7-4193-8649-2ba7854458b8\") " Oct 30 18:12:16.673095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:16.673026 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-openshift-ca\") pod \"973674e3-e3f7-4193-8649-2ba7854458b8\" (UID: \"973674e3-e3f7-4193-8649-2ba7854458b8\") " Oct 30 18:12:16.673481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:16.673451 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-openshift-ca" pod "973674e3-e3f7-4193-8649-2ba7854458b8" (UID: "973674e3-e3f7-4193-8649-2ba7854458b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:16.673629 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:16.673608 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:12:16.675233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:16.675201 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973674e3-e3f7-4193-8649-2ba7854458b8-kube-api-access-zhjgd" pod "973674e3-e3f7-4193-8649-2ba7854458b8" (UID: "973674e3-e3f7-4193-8649-2ba7854458b8"). InnerVolumeSpecName "kube-api-access-zhjgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:16.675558 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:16.675535 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-xks-ca" pod "973674e3-e3f7-4193-8649-2ba7854458b8" (UID: "973674e3-e3f7-4193-8649-2ba7854458b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:16.774880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:16.774784 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhjgd\" (UniqueName: \"kubernetes.io/projected/973674e3-e3f7-4193-8649-2ba7854458b8-kube-api-access-zhjgd\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:12:16.774880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:16.774820 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/973674e3-e3f7-4193-8649-2ba7854458b8-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:12:17.484538 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:17.484446 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" Oct 30 18:12:17.484701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:17.484447 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9" event={"ID":"973674e3-e3f7-4193-8649-2ba7854458b8","Type":"ContainerDied","Data":"41dbd4fdc5bb36bdda21fa0a92276b36089540afba1cf2b93d4d400a2ed01842"} Oct 30 18:12:17.484701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:17.484589 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41dbd4fdc5bb36bdda21fa0a92276b36089540afba1cf2b93d4d400a2ed01842" Oct 30 18:12:21.360061 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:21.360030 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:12:21.366095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:21.366067 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:12:21.366726 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:21.366700 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:12:21.378763 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:21.378731 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:12:21.439880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:21.439837 4883 scope.go:122] "RemoveContainer" containerID="12270eb7f549d2274574a4ffa13e517a17f70d61591497643dd2f5046e4de758" Oct 30 18:12:21.452761 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:21.452737 4883 scope.go:122] "RemoveContainer" containerID="c87a2dbfb14a4f91d0df1cf2a1ee25c8e2c91b17de9d03ad85fb6c5697f1a188" Oct 30 18:12:21.463492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:21.463464 4883 scope.go:122] "RemoveContainer" containerID="fd6ea32cdad4c430ff2cc6e44d50e51fd0d74d50ec1e1e6dd7970c789d34fa3e" Oct 30 18:12:26.156956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:26.156052 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:26.156956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:26.156142 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:26.251957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:26.251129 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:26.251957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:26.251198 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:26.677962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:26.675056 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:26.714967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:26.714037 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:27.067025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:27.065993 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k"] Oct 30 18:12:27.110957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:27.110086 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2536922-bt27k"] Oct 30 18:12:27.205959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:27.204977 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a8e400-b4f9-4d1b-ba69-73963dbc5adb" path="/var/lib/kubelet/pods/b4a8e400-b4f9-4d1b-ba69-73963dbc5adb/volumes" Oct 30 18:12:28.221961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:28.193659 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:28.221961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:28.195010 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:12:28.340506 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:28.336780 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:28.400485 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:28.396045 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:12:28.487361 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:28.487252 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:28.602401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:28.602352 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:12:28.627607 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:28.627561 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:12:28.691968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:28.677192 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:12:28.715364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:28.715316 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:28.916960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:28.909292 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:28.917428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:28.917368 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.097561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.097499 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.177018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.176075 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.243965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.224128 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.337101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.244554 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 30 18:12:29.337101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.244605 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.337101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.262405 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.341960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.337864 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.371873 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.357016 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Oct 30 18:12:29.396422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.396372 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.443466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.443367 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.450844 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.450758 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:12:29.450844 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.450832 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:12:29.451116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.450868 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:12:29.494961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.490096 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:12:29.604527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.604472 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:12:29.606060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.606027 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:12:29.635103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.635054 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:12:29.635723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.635697 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:12:29.689220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.689164 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:12:29.717133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.717008 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:12:29.717488 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.717447 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.737372 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.718135 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:12:29.807151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.807037 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded" Oct 30 18:12:29.941166 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.911114 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:29.941166 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.911568 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:12:29.996581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:29.996365 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:12:30.027848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.012313 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.027848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.013558 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:12:30.335161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.335054 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.336435 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.335677 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.373764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.336648 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:12:30.373764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.336714 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:12:30.373764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.373052 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.373764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.373457 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.419961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.410057 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.532962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.497429 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.580019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.551982 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Oct 30 18:12:30.580342 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.552044 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.581082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.581016 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:12:30.605967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.605461 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.626108 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.608397 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:30.626108 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.608462 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.647022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.641917 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.707101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.707044 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:30.724543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.724054 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:12:30.948856 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.948570 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:12:30.950360 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:30.950329 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:12:31.363963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.349069 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.381966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.374181 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.381966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.374637 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:12:31.385052 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.385015 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.389965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.385075 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:31.389965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.385901 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.411489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.411445 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.433101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.433067 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:12:31.620023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.611243 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:12:31.635138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.635070 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.689059 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.682425 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:31.764036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.763467 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:12:31.965025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:31.959116 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.018792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.018747 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.384469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.384406 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.415336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.387126 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.432011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.387201 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:12:32.432011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.388022 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:12:32.436431 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.434482 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.454093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.451417 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:12:32.549134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.549088 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:32.555082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.555028 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.622234 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.622181 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.640809 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.640677 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.697365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.697301 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:12:32.764748 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.763821 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:32.990213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:32.990068 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.049249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.027066 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.049249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.027614 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:12:33.414194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.414141 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.416713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.416674 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.417972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.417924 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:12:33.419126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.419097 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:12:33.465962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.465703 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.489960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.487517 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:12:33.489960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.488381 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:12:33.603972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.603594 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:33.603972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.603676 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.644114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.633472 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.654209 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.647050 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:33.711353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.711249 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:12:33.814053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.801836 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:12:33.972012 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:33.966816 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:12:34.061970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.045449 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:12:34.082106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.065521 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:12:34.082106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.068852 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:34.083545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.083519 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:12:34.091408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.085502 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:12:34.091408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.088250 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:12:34.146255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.146222 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:12:34.156714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.156686 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:12:34.190262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.189318 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:12:34.190262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.190166 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:12:34.190508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.190285 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:12:34.192335 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.191806 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:12:34.233609 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:34.233524 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:12:35.086788 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:12:35.086748 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:12:38.667672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.667631 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w"] Oct 30 18:12:38.668284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.668265 4883 cpu_manager.go:414] "containerMap: removing container" podUID="973674e3-e3f7-4193-8649-2ba7854458b8" containerName="revoke-keys" Oct 30 18:12:38.668284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.668283 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="973674e3-e3f7-4193-8649-2ba7854458b8" containerName="revoke-keys" Oct 30 18:12:38.668468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.668453 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="973674e3-e3f7-4193-8649-2ba7854458b8" containerName="revoke-keys" Oct 30 18:12:38.674040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.674013 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.690219 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.690189 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c5891f-openshift-default-dockercfg-hmphn\"" Oct 30 18:12:38.708546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.708505 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w"] Oct 30 18:12:38.795179 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.795132 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istiod-ca-cert\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.795398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.795262 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-envoy\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.795398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.795338 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-token\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.795508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.795489 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-data\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.795545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.795533 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-certs\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.795685 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.795665 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-credential-socket\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.795725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.795706 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-podinfo\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.795780 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.795762 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4dp6\" (UniqueName: \"kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-kube-api-access-f4dp6\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.795816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.795796 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-socket\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.897315 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.897277 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-credential-socket\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.897505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.897329 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-podinfo\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.897505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.897359 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4dp6\" (UniqueName: \"kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-kube-api-access-f4dp6\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.897505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.897380 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-socket\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.897505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.897403 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istiod-ca-cert\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.897505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.897434 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-envoy\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.897505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.897489 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-token\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.897812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.897574 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-data\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.897812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.897605 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-certs\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.897925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.897847 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-socket\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.898025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.897987 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-credential-socket\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.898103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.898078 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-certs\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.898279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.898255 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-data\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.898784 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.898760 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istiod-ca-cert\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.900161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.900130 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-envoy\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.900501 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.900398 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-podinfo\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.923835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.923752 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-token\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.923835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.923816 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4dp6\" (UniqueName: \"kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-kube-api-access-f4dp6\") pod \"e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:38.994270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:38.994234 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:39.347716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:39.347683 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w"] Oct 30 18:12:39.348117 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:12:39.348090 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd4bf78_d814_4cd9_80b2_730ae3eebaeb.slice/crio-89b0e3298e456d33e356f445b7e093edecca9c2c16bf63d3a6650c2c3257b8d7 WatchSource:0}: Error finding container 89b0e3298e456d33e356f445b7e093edecca9c2c16bf63d3a6650c2c3257b8d7: Status 404 returned error can't find the container with id 89b0e3298e456d33e356f445b7e093edecca9c2c16bf63d3a6650c2c3257b8d7 Oct 30 18:12:39.351144 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:39.351097 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:12:39.351310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:39.351257 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:12:39.351426 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:39.351334 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:12:39.641270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:39.641156 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" event={"ID":"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb","Type":"ContainerStarted","Data":"b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30"} Oct 30 18:12:39.641270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:39.641232 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" event={"ID":"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb","Type":"ContainerStarted","Data":"89b0e3298e456d33e356f445b7e093edecca9c2c16bf63d3a6650c2c3257b8d7"} Oct 30 18:12:39.690064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:39.690004 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" podStartSLOduration=1.689987671 podStartE2EDuration="1.689987671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:39.68807302 +0000 UTC m=+1218.645076869" watchObservedRunningTime="2026-10-30 18:12:39.689987671 +0000 UTC m=+1218.646991506" Oct 30 18:12:39.994512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:39.994421 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:40.996175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:40.996136 4883 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 18:12:40.996624 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:40.996218 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" podUID="7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 18:12:41.995061 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:41.995012 4883 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 18:12:41.995243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:41.995090 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" podUID="7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 18:12:42.994609 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:42.994568 4883 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 18:12:42.995073 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:42.994632 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" podUID="7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 18:12:44.000797 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:44.000320 4883 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 18:12:44.000797 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:44.000398 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" podUID="7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 18:12:44.994853 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:44.994807 4883 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 18:12:44.995206 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:44.994877 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" podUID="7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 18:12:45.999810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:45.999778 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:46.000334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:46.000220 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:12:46.001159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:12:46.001135 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:13:05.168463 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:13:05.168422 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:13:06.863585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:06.863547 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb"] Oct 30 18:13:06.868635 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:06.868611 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:06.885925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:06.885888 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:13:06.892139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:06.892101 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb"] Oct 30 18:13:06.899916 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:06.899884 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:13:06.900191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:06.899923 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:13:06.900191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:06.899957 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:13:06.963208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:06.963175 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b4b6c2-rljpb\" (UID: \"257f59ea-768d-4f76-b70f-e2a460559980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:06.963404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:06.963225 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b4b6c2-rljpb\" (UID: \"257f59ea-768d-4f76-b70f-e2a460559980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:06.963486 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:06.963457 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxqf\" (UniqueName: \"kubernetes.io/projected/257f59ea-768d-4f76-b70f-e2a460559980-kube-api-access-xhxqf\") pod \"maas-api-revoke-sub-48b4b6c2-rljpb\" (UID: \"257f59ea-768d-4f76-b70f-e2a460559980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:07.065089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:07.065044 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxqf\" (UniqueName: \"kubernetes.io/projected/257f59ea-768d-4f76-b70f-e2a460559980-kube-api-access-xhxqf\") pod \"maas-api-revoke-sub-48b4b6c2-rljpb\" (UID: \"257f59ea-768d-4f76-b70f-e2a460559980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:07.065303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:07.065278 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b4b6c2-rljpb\" (UID: \"257f59ea-768d-4f76-b70f-e2a460559980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:07.065371 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:07.065330 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b4b6c2-rljpb\" (UID: \"257f59ea-768d-4f76-b70f-e2a460559980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:07.066163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:07.066135 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b4b6c2-rljpb\" (UID: \"257f59ea-768d-4f76-b70f-e2a460559980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:07.067927 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:07.067895 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b4b6c2-rljpb\" (UID: \"257f59ea-768d-4f76-b70f-e2a460559980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:07.091308 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:07.091273 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxqf\" (UniqueName: \"kubernetes.io/projected/257f59ea-768d-4f76-b70f-e2a460559980-kube-api-access-xhxqf\") pod \"maas-api-revoke-sub-48b4b6c2-rljpb\" (UID: \"257f59ea-768d-4f76-b70f-e2a460559980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:07.184124 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:07.184031 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:07.500251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:07.500218 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb"] Oct 30 18:13:07.500629 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:13:07.500608 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257f59ea_768d_4f76_b70f_e2a460559980.slice/crio-a72fd53ae013851def373493a1a7f847eba7b179000351be45a2647843ae9569 WatchSource:0}: Error finding container a72fd53ae013851def373493a1a7f847eba7b179000351be45a2647843ae9569: Status 404 returned error can't find the container with id a72fd53ae013851def373493a1a7f847eba7b179000351be45a2647843ae9569 Oct 30 18:13:07.938785 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:07.938745 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" event={"ID":"257f59ea-768d-4f76-b70f-e2a460559980","Type":"ContainerStarted","Data":"95a8b6a2b2b0cfc75cd56af332a770a96fecd506a2d6a2fb8682e5023444febd"} Oct 30 18:13:07.939202 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:07.938804 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" event={"ID":"257f59ea-768d-4f76-b70f-e2a460559980","Type":"ContainerStarted","Data":"a72fd53ae013851def373493a1a7f847eba7b179000351be45a2647843ae9569"} Oct 30 18:13:07.981779 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:07.981726 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" podStartSLOduration=1.981708399 podStartE2EDuration="1.981708399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:07.981254517 +0000 UTC m=+1246.938258356" watchObservedRunningTime="2026-10-30 18:13:07.981708399 +0000 UTC m=+1246.938712236" Oct 30 18:13:08.948470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:08.948433 4883 generic.go:356] "Generic (PLEG): container finished" podID="257f59ea-768d-4f76-b70f-e2a460559980" containerID="95a8b6a2b2b0cfc75cd56af332a770a96fecd506a2d6a2fb8682e5023444febd" exitCode=0 Oct 30 18:13:08.948877 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:08.948479 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" event={"ID":"257f59ea-768d-4f76-b70f-e2a460559980","Type":"ContainerDied","Data":"95a8b6a2b2b0cfc75cd56af332a770a96fecd506a2d6a2fb8682e5023444febd"} Oct 30 18:13:10.265313 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:10.264467 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:10.356749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:10.356719 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257f59ea-768d-4f76-b70f-e2a460559980-kube-api-access-xhxqf\" (UniqueName: \"kubernetes.io/projected/257f59ea-768d-4f76-b70f-e2a460559980-kube-api-access-xhxqf\") pod \"257f59ea-768d-4f76-b70f-e2a460559980\" (UID: \"257f59ea-768d-4f76-b70f-e2a460559980\") " Oct 30 18:13:10.357049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:10.357025 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-xks-ca\") pod \"257f59ea-768d-4f76-b70f-e2a460559980\" (UID: \"257f59ea-768d-4f76-b70f-e2a460559980\") " Oct 30 18:13:10.357382 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:10.357361 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-openshift-ca\") pod \"257f59ea-768d-4f76-b70f-e2a460559980\" (UID: \"257f59ea-768d-4f76-b70f-e2a460559980\") " Oct 30 18:13:10.357993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:10.357923 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-openshift-ca" pod "257f59ea-768d-4f76-b70f-e2a460559980" (UID: "257f59ea-768d-4f76-b70f-e2a460559980"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:10.358295 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:10.358280 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:10.365692 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:10.362634 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257f59ea-768d-4f76-b70f-e2a460559980-kube-api-access-xhxqf" pod "257f59ea-768d-4f76-b70f-e2a460559980" (UID: "257f59ea-768d-4f76-b70f-e2a460559980"). InnerVolumeSpecName "kube-api-access-xhxqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:10.366298 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:10.366262 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-xks-ca" pod "257f59ea-768d-4f76-b70f-e2a460559980" (UID: "257f59ea-768d-4f76-b70f-e2a460559980"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:10.459483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:10.459442 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhxqf\" (UniqueName: \"kubernetes.io/projected/257f59ea-768d-4f76-b70f-e2a460559980-kube-api-access-xhxqf\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:10.459483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:10.459473 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257f59ea-768d-4f76-b70f-e2a460559980-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:11.031387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:11.029985 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" event={"ID":"257f59ea-768d-4f76-b70f-e2a460559980","Type":"ContainerDied","Data":"a72fd53ae013851def373493a1a7f847eba7b179000351be45a2647843ae9569"} Oct 30 18:13:11.031387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:11.030051 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a72fd53ae013851def373493a1a7f847eba7b179000351be45a2647843ae9569" Oct 30 18:13:11.031387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:11.030124 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb" Oct 30 18:13:18.152576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.152535 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t"] Oct 30 18:13:18.153245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.153190 4883 cpu_manager.go:414] "containerMap: removing container" podUID="257f59ea-768d-4f76-b70f-e2a460559980" containerName="revoke-keys" Oct 30 18:13:18.153245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.153205 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257f59ea-768d-4f76-b70f-e2a460559980" containerName="revoke-keys" Oct 30 18:13:18.153374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.153323 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="257f59ea-768d-4f76-b70f-e2a460559980" containerName="revoke-keys" Oct 30 18:13:18.157572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.157545 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.178476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.178438 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t"] Oct 30 18:13:18.186584 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.186547 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c6b2120f\"/\"openshift-service-ca.crt\"" Oct 30 18:13:18.188004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.187974 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c6b2120f\"/\"e2e-lifecycle-model-c6b2120f-kserve-self-signed-certs\"" Oct 30 18:13:18.188004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.187982 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c6b2120f\"/\"default-dockercfg-6q6fc\"" Oct 30 18:13:18.188229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.187926 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c6b2120f\"/\"kube-root-ca.crt\"" Oct 30 18:13:18.273735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.273696 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-model-cache\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.273971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.273760 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tls-certs\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.273971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.273896 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-home\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.273971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.273957 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-dshm\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.274154 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.274088 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tmp-dir\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.274154 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.274117 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgm22\" (UniqueName: \"kubernetes.io/projected/57ec35a1-8960-44ec-9fb3-eb9df9492b92-kube-api-access-fgm22\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.375588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.375540 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tmp-dir\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.375588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.375592 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgm22\" (UniqueName: \"kubernetes.io/projected/57ec35a1-8960-44ec-9fb3-eb9df9492b92-kube-api-access-fgm22\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.375868 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.375707 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-model-cache\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.375868 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.375770 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tls-certs\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.375868 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.375859 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-home\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.376056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.375901 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-dshm\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.376157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.376131 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-model-cache\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.376295 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.376216 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-home\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.376368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.376336 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tmp-dir\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.378328 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.378297 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-dshm\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.378919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.378896 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tls-certs\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.400286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.400247 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgm22\" (UniqueName: \"kubernetes.io/projected/57ec35a1-8960-44ec-9fb3-eb9df9492b92-kube-api-access-fgm22\") pod \"e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.478786 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.478429 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:18.871722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:18.871688 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t"] Oct 30 18:13:18.872721 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:13:18.872692 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57ec35a1_8960_44ec_9fb3_eb9df9492b92.slice/crio-51050b7975f53963d2e20fd41dcd23c7173dc51b1225d51fdb54b1874918b726 WatchSource:0}: Error finding container 51050b7975f53963d2e20fd41dcd23c7173dc51b1225d51fdb54b1874918b726: Status 404 returned error can't find the container with id 51050b7975f53963d2e20fd41dcd23c7173dc51b1225d51fdb54b1874918b726 Oct 30 18:13:19.101499 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:19.101460 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" event={"ID":"57ec35a1-8960-44ec-9fb3-eb9df9492b92","Type":"ContainerStarted","Data":"6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35"} Oct 30 18:13:19.101499 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:19.101515 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" event={"ID":"57ec35a1-8960-44ec-9fb3-eb9df9492b92","Type":"ContainerStarted","Data":"51050b7975f53963d2e20fd41dcd23c7173dc51b1225d51fdb54b1874918b726"} Oct 30 18:13:19.148220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:19.148109 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podStartSLOduration=1.14808723 podStartE2EDuration="1.14808723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:19.146115344 +0000 UTC m=+1258.103119181" watchObservedRunningTime="2026-10-30 18:13:19.14808723 +0000 UTC m=+1258.105091104" Oct 30 18:13:19.480117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:19.480027 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:21.505411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:21.505375 4883 scope.go:122] "RemoveContainer" containerID="2449edce47acdaf79a7a220c4732a53faee639595d481ab9cb5014e03ed28842" Oct 30 18:13:21.517152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:21.517125 4883 scope.go:122] "RemoveContainer" containerID="045585b5100006868d9889226dd3823a8a0ead2de6fae25d1808d89de6a882c6" Oct 30 18:13:21.527914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:21.527885 4883 scope.go:122] "RemoveContainer" containerID="0a82d1e39bc741a9b5cd62a18f808f7bc0cf6248e6af6e61c8c57b94e64dc104" Oct 30 18:13:21.541312 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:21.541288 4883 scope.go:122] "RemoveContainer" containerID="eaf1c99a76cd5302c7d8466821af1ddc0b3589b3b5044d33eca10a3c96f3a63d" Oct 30 18:13:25.757183 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:25.757144 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x"] Oct 30 18:13:25.757633 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:25.757353 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" podUID="18290ea1-6092-4b79-b48e-40121387ad50" containerName="istio-proxy" containerID="cri-o://4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48" gracePeriod=30 Oct 30 18:13:28.479449 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:28.479414 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:28.496255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:28.496211 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:29.188866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:29.188832 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:31.061102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.061071 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 18:13:31.138043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.138005 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18290ea1-6092-4b79-b48e-40121387ad50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18290ea1-6092-4b79-b48e-40121387ad50-istio-podinfo\") pod \"18290ea1-6092-4b79-b48e-40121387ad50\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " Oct 30 18:13:31.138241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.138070 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-envoy\") pod \"18290ea1-6092-4b79-b48e-40121387ad50\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " Oct 30 18:13:31.138241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.138223 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-certs\") pod \"18290ea1-6092-4b79-b48e-40121387ad50\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " Oct 30 18:13:31.138332 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.138249 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18290ea1-6092-4b79-b48e-40121387ad50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18290ea1-6092-4b79-b48e-40121387ad50-istiod-ca-cert\") pod \"18290ea1-6092-4b79-b48e-40121387ad50\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " Oct 30 18:13:31.138332 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.138277 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-data\") pod \"18290ea1-6092-4b79-b48e-40121387ad50\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " Oct 30 18:13:31.138332 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.138320 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-credential-socket\") pod \"18290ea1-6092-4b79-b48e-40121387ad50\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " Oct 30 18:13:31.138476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.138384 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-socket\") pod \"18290ea1-6092-4b79-b48e-40121387ad50\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " Oct 30 18:13:31.138476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.138428 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-kube-api-access-5l6z5\" (UniqueName: \"kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-kube-api-access-5l6z5\") pod \"18290ea1-6092-4b79-b48e-40121387ad50\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " Oct 30 18:13:31.138575 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.138501 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-istio-token\" (UniqueName: \"kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-istio-token\") pod \"18290ea1-6092-4b79-b48e-40121387ad50\" (UID: \"18290ea1-6092-4b79-b48e-40121387ad50\") " Oct 30 18:13:31.139602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.139569 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-data" pod "18290ea1-6092-4b79-b48e-40121387ad50" (UID: "18290ea1-6092-4b79-b48e-40121387ad50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:31.139791 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.139665 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18290ea1-6092-4b79-b48e-40121387ad50-istiod-ca-cert" pod "18290ea1-6092-4b79-b48e-40121387ad50" (UID: "18290ea1-6092-4b79-b48e-40121387ad50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:31.140084 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.140038 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-credential-socket" pod "18290ea1-6092-4b79-b48e-40121387ad50" (UID: "18290ea1-6092-4b79-b48e-40121387ad50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:31.140359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.140332 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-certs" pod "18290ea1-6092-4b79-b48e-40121387ad50" (UID: "18290ea1-6092-4b79-b48e-40121387ad50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:31.141721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.141075 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18290ea1-6092-4b79-b48e-40121387ad50-istio-podinfo" pod "18290ea1-6092-4b79-b48e-40121387ad50" (UID: "18290ea1-6092-4b79-b48e-40121387ad50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:13:31.141721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.141286 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-socket" pod "18290ea1-6092-4b79-b48e-40121387ad50" (UID: "18290ea1-6092-4b79-b48e-40121387ad50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:31.143792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.143760 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-envoy" pod "18290ea1-6092-4b79-b48e-40121387ad50" (UID: "18290ea1-6092-4b79-b48e-40121387ad50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:31.144014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.143915 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-istio-token" pod "18290ea1-6092-4b79-b48e-40121387ad50" (UID: "18290ea1-6092-4b79-b48e-40121387ad50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:31.144411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.144384 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-kube-api-access-5l6z5" pod "18290ea1-6092-4b79-b48e-40121387ad50" (UID: "18290ea1-6092-4b79-b48e-40121387ad50"). InnerVolumeSpecName "kube-api-access-5l6z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:31.213644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.213608 4883 generic.go:356] "Generic (PLEG): container finished" podID="18290ea1-6092-4b79-b48e-40121387ad50" containerID="4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48" exitCode=0 Oct 30 18:13:31.213834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.213670 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" Oct 30 18:13:31.213834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.213704 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" event={"ID":"18290ea1-6092-4b79-b48e-40121387ad50","Type":"ContainerDied","Data":"4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48"} Oct 30 18:13:31.213834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.213801 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x" event={"ID":"18290ea1-6092-4b79-b48e-40121387ad50","Type":"ContainerDied","Data":"bf76e315f0150e7cd015d947f6c7874d155aa4c2d55be6cf20ef2c2854fa066f"} Oct 30 18:13:31.213834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.213828 4883 scope.go:122] "RemoveContainer" containerID="4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48" Oct 30 18:13:31.225438 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.225417 4883 scope.go:122] "RemoveContainer" containerID="4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48" Oct 30 18:13:31.225797 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:13:31.225775 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48\": container with ID starting with 4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48 not found: ID does not exist" containerID="4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48" Oct 30 18:13:31.225836 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.225810 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48"} err="failed to get container status \"4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48\": rpc error: code = NotFound desc = could not find container \"4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48\": container with ID starting with 4080b3e6b0746df5f73031c02bac59ef0a0ea2b2b4a7f5bbb4e7be4e98860d48 not found: ID does not exist" Oct 30 18:13:31.241641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.241606 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:31.241986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.241886 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:31.241986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.241906 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l6z5\" (UniqueName: \"kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-kube-api-access-5l6z5\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:31.241986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.241920 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18290ea1-6092-4b79-b48e-40121387ad50-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:31.241986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.241985 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18290ea1-6092-4b79-b48e-40121387ad50-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:31.242304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.241999 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:31.242304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.242013 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:31.242304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.242026 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18290ea1-6092-4b79-b48e-40121387ad50-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:31.242304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.242038 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18290ea1-6092-4b79-b48e-40121387ad50-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:31.280141 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.280103 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x"] Oct 30 18:13:31.284682 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:31.284637 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-p8z8x"] Oct 30 18:13:33.255960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:33.235704 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18290ea1-6092-4b79-b48e-40121387ad50" path="/var/lib/kubelet/pods/18290ea1-6092-4b79-b48e-40121387ad50/volumes" Oct 30 18:13:34.124015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:34.114099 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.190957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:34.188046 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.431016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:34.430045 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:13:34.504240 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:34.503034 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:13:34.649960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:34.645034 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.664864 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:34.664014 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.698387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:34.697032 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.770241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:34.768043 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.770882 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:34.770557 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:34.770882 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:34.770620 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:34.867963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:34.856826 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:34.867963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:34.856907 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.000167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.000040 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.090252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.090183 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.091253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.091218 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.119094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.119052 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.134963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.119679 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:13:35.356032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.341138 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:13:35.397640 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:13:35.370543 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:13:35.397904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.380076 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:13:35.438087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.381690 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.438087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.436953 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:13:35.511955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.511892 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.608021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.583408 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.669997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.650637 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:35.694962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.670310 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:13:35.696370 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.696340 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:13:35.696947 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.695274 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:35.697732 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.697713 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:13:36.015020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.014329 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:36.046951 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.650917 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:36.047270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:35.977630 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:13:36.096276 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.096227 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:36.176220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.172922 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:13:36.176220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.173238 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:13:36.176220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.173456 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:13:36.176220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.173511 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:13:36.176220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.174741 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:13:36.239508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.239469 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:13:36.364030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.342367 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:36.364030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.351323 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:13:36.364030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.360665 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:13:36.399408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.399359 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:13:36.461114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.458889 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:13:36.489963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.463011 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:13:36.518642 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.482525 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:36.542167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.520810 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:13:36.542641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.542609 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:13:36.754155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.754042 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:36.837962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.834180 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.071021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.016415 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr"] Oct 30 18:13:37.072192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.072153 4883 cpu_manager.go:414] "containerMap: removing container" podUID="18290ea1-6092-4b79-b48e-40121387ad50" containerName="istio-proxy" Oct 30 18:13:37.072393 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.072376 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18290ea1-6092-4b79-b48e-40121387ad50" containerName="istio-proxy" Oct 30 18:13:37.075956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.072647 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="18290ea1-6092-4b79-b48e-40121387ad50" containerName="istio-proxy" Oct 30 18:13:37.127638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:36.879921 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 30 18:13:37.209984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.135153 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.209984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.135383 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr"] Oct 30 18:13:37.231961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.135485 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.231961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.146141 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.231961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.146825 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.246961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.146867 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:13:37.246961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.245471 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.246961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.246898 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded" Oct 30 18:13:37.373000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.365150 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.373000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.366587 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:13:37.373000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.366651 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:13:37.373000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.366779 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-44kjj\"" Oct 30 18:13:37.373000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.366832 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:13:37.373000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.369007 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.402960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.397087 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.402960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.397137 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb622\" (UniqueName: \"kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-kube-api-access-tb622\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.402960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.397208 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49a4773-1070-4484-9894-32d306f96e59-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.402960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.397295 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.402960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.397384 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.402960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.397429 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.402960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.397462 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.418097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.418043 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.514958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.510342 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.514958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.510424 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.514958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.510460 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.514958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.510608 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.514958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.510634 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb622\" (UniqueName: \"kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-kube-api-access-tb622\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.514958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.510717 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49a4773-1070-4484-9894-32d306f96e59-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.514958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.510801 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.514958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.511841 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:13:37.521222 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.521065 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:13:37.574586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.574216 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:13:37.602982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.601582 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.648013 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.636121 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.660959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.654256 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49a4773-1070-4484-9894-32d306f96e59-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.672589 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.672518 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.682969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.681393 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.753962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.726169 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.884668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.884632 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb622\" (UniqueName: \"kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-kube-api-access-tb622\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47vdr\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:37.979021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.978196 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:37.979021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:37.978269 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.020078 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.006627 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:38.023129 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.023074 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:38.023487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.023383 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.115495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.114112 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:13:38.115495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.114550 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:13:38.202896 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.202841 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.232177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.232067 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:38.232177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.232146 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.232413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.232228 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:13:38.232615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.232578 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:13:38.315960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.269264 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.316699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.271017 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:13:38.320959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.271055 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:13:38.371320 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.371269 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.396318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.394604 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.421965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.418747 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.436799 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.436754 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.571024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.530093 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:13:38.599169 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.573845 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.636037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.599490 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:13:38.645973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:38.645037 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.163975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.156630 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:13:39.196139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.184198 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:13:39.218019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.215057 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.235191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.230708 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.258974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.257128 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:13:39.287962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.261863 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:13:39.287962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.282298 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.340663 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.340550 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:13:39.341326 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.341294 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:13:39.353893 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.353841 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.400994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.398643 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.400994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.399076 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:13:39.419127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.419060 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.514039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.508520 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.650222 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.649061 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.650743 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.650532 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:13:39.651094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.650892 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:13:39.775960 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:13:39.749296 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49a4773_1070_4484_9894_32d306f96e59.slice/crio-4f82a49ab009f8bdea05f2d219d967720effb4790183e65cb3c2a81255fe5133 WatchSource:0}: Error finding container 4f82a49ab009f8bdea05f2d219d967720effb4790183e65cb3c2a81255fe5133: Status 404 returned error can't find the container with id 4f82a49ab009f8bdea05f2d219d967720effb4790183e65cb3c2a81255fe5133 Oct 30 18:13:39.775960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.749546 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.937997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:39.937475 4883 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:13:40.069822 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.069780 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Oct 30 18:13:40.209217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.209085 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.225314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.225255 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:13:40.249570 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.249513 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.343961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.259050 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.354974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.279220 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.354974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.286524 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn"] Oct 30 18:13:40.354974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.351716 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95fe591e-kv7dn"] Oct 30 18:13:40.354974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.351811 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.354974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.351928 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.399960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.399707 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.414962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.400194 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:13:40.434100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.434035 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.543035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.537716 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr"] Oct 30 18:13:40.551376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.544374 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.558617 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.544409 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Oct 30 18:13:40.660005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.653513 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.660005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.654882 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:13:40.660005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:40.655234 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:13:41.011959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.010033 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" event={"ID":"a49a4773-1070-4484-9894-32d306f96e59","Type":"ContainerStarted","Data":"4f82a49ab009f8bdea05f2d219d967720effb4790183e65cb3c2a81255fe5133"} Oct 30 18:13:41.202148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.201080 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c78665-d28d-46c8-9e17-d5f4b4943319" path="/var/lib/kubelet/pods/49c78665-d28d-46c8-9e17-d5f4b4943319/volumes" Oct 30 18:13:41.211153 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.211094 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.212122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.212093 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:13:41.227203 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.227146 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:13:41.256161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.256110 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.282265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.282168 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:13:41.354954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.345043 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:13:41.354954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.353206 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.361071 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.355077 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:13:41.364389 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.361808 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:13:41.414365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.414334 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:13:41.455236 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.455174 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:13:41.474902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.474861 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:13:41.521953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.495101 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:13:41.540150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.535377 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.605260 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.605206 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:13:41.605915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.605891 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:13:41.609272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.608566 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:13:41.614840 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.614798 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:13:41.620969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.619207 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:13:41.621646 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.621618 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:13:41.625225 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.625184 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:13:41.627516 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.627461 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:13:41.628732 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:41.628711 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:13:43.726579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:43.726538 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:13:43.727029 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:43.726632 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:13:44.067035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:44.066898 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" event={"ID":"a49a4773-1070-4484-9894-32d306f96e59","Type":"ContainerStarted","Data":"4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f"} Oct 30 18:13:44.067297 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:44.067268 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:44.068924 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:44.068889 4883 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-47vdr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:13:44.069079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:44.068990 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" podUID="a49a4773-1070-4484-9894-32d306f96e59" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:13:44.142320 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:44.142230 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" podStartSLOduration=5.355210507 podStartE2EDuration="9.142211767s" totalImagesPullingTime="3.78700126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:44.128603455 +0000 UTC m=+1283.085607290" watchObservedRunningTime="2026-10-30 18:13:44.142211767 +0000 UTC m=+1283.099215603" Oct 30 18:13:45.103178 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.103141 4883 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-47vdr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:13:45.103666 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.103195 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" podUID="a49a4773-1070-4484-9894-32d306f96e59" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:13:45.232114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.232069 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x"] Oct 30 18:13:45.237708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.237678 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:45.267859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.266552 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:13:45.267859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.267205 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:13:45.267859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.267264 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:13:45.267859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.267323 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:13:45.270338 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.270290 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x"] Oct 30 18:13:45.298582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.295318 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83de5561-ndd8x\" (UID: \"34a11e90-5324-4af4-9555-151467c3c2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:45.298582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.295528 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x8z8\" (UniqueName: \"kubernetes.io/projected/34a11e90-5324-4af4-9555-151467c3c2d4-kube-api-access-4x8z8\") pod \"maas-api-revoke-sub-83de5561-ndd8x\" (UID: \"34a11e90-5324-4af4-9555-151467c3c2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:45.298582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.295591 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83de5561-ndd8x\" (UID: \"34a11e90-5324-4af4-9555-151467c3c2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:45.397002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.396778 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83de5561-ndd8x\" (UID: \"34a11e90-5324-4af4-9555-151467c3c2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:45.397002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.396954 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8z8\" (UniqueName: \"kubernetes.io/projected/34a11e90-5324-4af4-9555-151467c3c2d4-kube-api-access-4x8z8\") pod \"maas-api-revoke-sub-83de5561-ndd8x\" (UID: \"34a11e90-5324-4af4-9555-151467c3c2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:45.397293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.397020 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83de5561-ndd8x\" (UID: \"34a11e90-5324-4af4-9555-151467c3c2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:45.399408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.398067 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83de5561-ndd8x\" (UID: \"34a11e90-5324-4af4-9555-151467c3c2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:45.400351 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.400321 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83de5561-ndd8x\" (UID: \"34a11e90-5324-4af4-9555-151467c3c2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:45.431953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.430926 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8z8\" (UniqueName: \"kubernetes.io/projected/34a11e90-5324-4af4-9555-151467c3c2d4-kube-api-access-4x8z8\") pod \"maas-api-revoke-sub-83de5561-ndd8x\" (UID: \"34a11e90-5324-4af4-9555-151467c3c2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:45.572181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.571377 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:45.893054 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:45.890863 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x"] Oct 30 18:13:45.900224 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:13:45.900189 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a11e90_5324_4af4_9555_151467c3c2d4.slice/crio-46183f968f93ab5b55d00a57f370293472c5adfbe20d52a5a3ae6601304f11c5 WatchSource:0}: Error finding container 46183f968f93ab5b55d00a57f370293472c5adfbe20d52a5a3ae6601304f11c5: Status 404 returned error can't find the container with id 46183f968f93ab5b55d00a57f370293472c5adfbe20d52a5a3ae6601304f11c5 Oct 30 18:13:46.122980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:46.118284 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" event={"ID":"34a11e90-5324-4af4-9555-151467c3c2d4","Type":"ContainerStarted","Data":"46183f968f93ab5b55d00a57f370293472c5adfbe20d52a5a3ae6601304f11c5"} Oct 30 18:13:47.127761 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:47.127671 4883 generic.go:356] "Generic (PLEG): container finished" podID="34a11e90-5324-4af4-9555-151467c3c2d4" containerID="20bfdea67675d12f0b88763effa7da352aa343c47845919594b615726aaec628" exitCode=0 Oct 30 18:13:47.128136 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:47.127750 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" event={"ID":"34a11e90-5324-4af4-9555-151467c3c2d4","Type":"ContainerDied","Data":"20bfdea67675d12f0b88763effa7da352aa343c47845919594b615726aaec628"} Oct 30 18:13:48.103351 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:48.103323 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:13:48.337009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:48.336977 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:48.452072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:48.451925 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-openshift-ca\") pod \"34a11e90-5324-4af4-9555-151467c3c2d4\" (UID: \"34a11e90-5324-4af4-9555-151467c3c2d4\") " Oct 30 18:13:48.452237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:48.452146 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a11e90-5324-4af4-9555-151467c3c2d4-kube-api-access-4x8z8\" (UniqueName: \"kubernetes.io/projected/34a11e90-5324-4af4-9555-151467c3c2d4-kube-api-access-4x8z8\") pod \"34a11e90-5324-4af4-9555-151467c3c2d4\" (UID: \"34a11e90-5324-4af4-9555-151467c3c2d4\") " Oct 30 18:13:48.452237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:48.452204 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-xks-ca\") pod \"34a11e90-5324-4af4-9555-151467c3c2d4\" (UID: \"34a11e90-5324-4af4-9555-151467c3c2d4\") " Oct 30 18:13:48.452350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:48.452312 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-openshift-ca" pod "34a11e90-5324-4af4-9555-151467c3c2d4" (UID: "34a11e90-5324-4af4-9555-151467c3c2d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:48.452875 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:48.452852 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:48.454473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:48.454442 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-xks-ca" pod "34a11e90-5324-4af4-9555-151467c3c2d4" (UID: "34a11e90-5324-4af4-9555-151467c3c2d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:48.454567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:48.454451 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a11e90-5324-4af4-9555-151467c3c2d4-kube-api-access-4x8z8" pod "34a11e90-5324-4af4-9555-151467c3c2d4" (UID: "34a11e90-5324-4af4-9555-151467c3c2d4"). InnerVolumeSpecName "kube-api-access-4x8z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:48.554567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:48.554525 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x8z8\" (UniqueName: \"kubernetes.io/projected/34a11e90-5324-4af4-9555-151467c3c2d4-kube-api-access-4x8z8\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:48.554567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:48.554572 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a11e90-5324-4af4-9555-151467c3c2d4-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:49.167035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:49.166987 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" Oct 30 18:13:49.167035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:49.167004 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x" event={"ID":"34a11e90-5324-4af4-9555-151467c3c2d4","Type":"ContainerDied","Data":"46183f968f93ab5b55d00a57f370293472c5adfbe20d52a5a3ae6601304f11c5"} Oct 30 18:13:49.167409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:49.167104 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46183f968f93ab5b55d00a57f370293472c5adfbe20d52a5a3ae6601304f11c5" Oct 30 18:13:54.033869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:54.033827 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz"] Oct 30 18:13:54.046908 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:54.046862 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282922cb-fcnjz"] Oct 30 18:13:55.189111 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:55.189072 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8ecd72-c079-440b-811a-903809ad700e" path="/var/lib/kubelet/pods/8a8ecd72-c079-440b-811a-903809ad700e/volumes" Oct 30 18:13:55.940702 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:55.940660 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl"] Oct 30 18:13:55.941374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:55.941350 4883 cpu_manager.go:414] "containerMap: removing container" podUID="34a11e90-5324-4af4-9555-151467c3c2d4" containerName="revoke-keys" Oct 30 18:13:55.941374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:55.941375 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a11e90-5324-4af4-9555-151467c3c2d4" containerName="revoke-keys" Oct 30 18:13:55.941583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:55.941569 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a11e90-5324-4af4-9555-151467c3c2d4" containerName="revoke-keys" Oct 30 18:13:55.946689 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:55.946661 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:13:55.965969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:55.965910 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl"] Oct 30 18:13:55.975200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:55.975170 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:13:55.975436 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:55.975253 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:13:55.975436 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:55.975350 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:13:55.975555 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:55.975393 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:13:56.079954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:56.079900 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5891f-vnssl\" (UID: \"d0c71a7a-8287-4207-81bf-60be096b96d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:13:56.080149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:56.080073 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-277qk\" (UniqueName: \"kubernetes.io/projected/d0c71a7a-8287-4207-81bf-60be096b96d7-kube-api-access-277qk\") pod \"maas-api-revoke-keys-e2e-api-a-c5891f-vnssl\" (UID: \"d0c71a7a-8287-4207-81bf-60be096b96d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:13:56.080282 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:56.080259 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5891f-vnssl\" (UID: \"d0c71a7a-8287-4207-81bf-60be096b96d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:13:56.181685 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:56.181645 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5891f-vnssl\" (UID: \"d0c71a7a-8287-4207-81bf-60be096b96d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:13:56.181921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:56.181819 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5891f-vnssl\" (UID: \"d0c71a7a-8287-4207-81bf-60be096b96d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:13:56.181921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:56.181906 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-277qk\" (UniqueName: \"kubernetes.io/projected/d0c71a7a-8287-4207-81bf-60be096b96d7-kube-api-access-277qk\") pod \"maas-api-revoke-keys-e2e-api-a-c5891f-vnssl\" (UID: \"d0c71a7a-8287-4207-81bf-60be096b96d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:13:56.182722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:56.182688 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5891f-vnssl\" (UID: \"d0c71a7a-8287-4207-81bf-60be096b96d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:13:56.184534 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:56.184503 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c5891f-vnssl\" (UID: \"d0c71a7a-8287-4207-81bf-60be096b96d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:13:56.213253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:56.213165 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-277qk\" (UniqueName: \"kubernetes.io/projected/d0c71a7a-8287-4207-81bf-60be096b96d7-kube-api-access-277qk\") pod \"maas-api-revoke-keys-e2e-api-a-c5891f-vnssl\" (UID: \"d0c71a7a-8287-4207-81bf-60be096b96d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:13:56.261187 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:56.261148 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:13:56.582176 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:56.582136 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl"] Oct 30 18:13:56.582712 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:13:56.582688 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c71a7a_8287_4207_81bf_60be096b96d7.slice/crio-b368afd7154c8ca7e6d6aef2aec5ff7bc81ae8b2c93ec251e1c0984410da2b52 WatchSource:0}: Error finding container b368afd7154c8ca7e6d6aef2aec5ff7bc81ae8b2c93ec251e1c0984410da2b52: Status 404 returned error can't find the container with id b368afd7154c8ca7e6d6aef2aec5ff7bc81ae8b2c93ec251e1c0984410da2b52 Oct 30 18:13:57.231636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:57.231543 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" event={"ID":"d0c71a7a-8287-4207-81bf-60be096b96d7","Type":"ContainerStarted","Data":"52064480904fc5871d78deabef5b8aecab38eb9981ca23c730b39d1d6fdf6add"} Oct 30 18:13:57.231636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:57.231605 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" event={"ID":"d0c71a7a-8287-4207-81bf-60be096b96d7","Type":"ContainerStarted","Data":"b368afd7154c8ca7e6d6aef2aec5ff7bc81ae8b2c93ec251e1c0984410da2b52"} Oct 30 18:13:57.281336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:57.281279 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" podStartSLOduration=2.281265477 podStartE2EDuration="2.281265477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:57.280989027 +0000 UTC m=+1296.237992863" watchObservedRunningTime="2026-10-30 18:13:57.281265477 +0000 UTC m=+1296.238269312" Oct 30 18:13:58.243781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:58.243738 4883 generic.go:356] "Generic (PLEG): container finished" podID="d0c71a7a-8287-4207-81bf-60be096b96d7" containerID="52064480904fc5871d78deabef5b8aecab38eb9981ca23c730b39d1d6fdf6add" exitCode=0 Oct 30 18:13:58.244355 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:58.243841 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" event={"ID":"d0c71a7a-8287-4207-81bf-60be096b96d7","Type":"ContainerDied","Data":"52064480904fc5871d78deabef5b8aecab38eb9981ca23c730b39d1d6fdf6add"} Oct 30 18:13:59.417694 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:59.417664 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:13:59.431758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:59.431725 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-xks-ca\") pod \"d0c71a7a-8287-4207-81bf-60be096b96d7\" (UID: \"d0c71a7a-8287-4207-81bf-60be096b96d7\") " Oct 30 18:13:59.431927 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:59.431820 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-openshift-ca\") pod \"d0c71a7a-8287-4207-81bf-60be096b96d7\" (UID: \"d0c71a7a-8287-4207-81bf-60be096b96d7\") " Oct 30 18:13:59.432103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:59.432084 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c71a7a-8287-4207-81bf-60be096b96d7-kube-api-access-277qk\" (UniqueName: \"kubernetes.io/projected/d0c71a7a-8287-4207-81bf-60be096b96d7-kube-api-access-277qk\") pod \"d0c71a7a-8287-4207-81bf-60be096b96d7\" (UID: \"d0c71a7a-8287-4207-81bf-60be096b96d7\") " Oct 30 18:13:59.432270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:59.432242 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-openshift-ca" pod "d0c71a7a-8287-4207-81bf-60be096b96d7" (UID: "d0c71a7a-8287-4207-81bf-60be096b96d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:59.432551 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:59.432534 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:59.434331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:59.434298 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-xks-ca" pod "d0c71a7a-8287-4207-81bf-60be096b96d7" (UID: "d0c71a7a-8287-4207-81bf-60be096b96d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:59.434461 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:59.434404 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c71a7a-8287-4207-81bf-60be096b96d7-kube-api-access-277qk" pod "d0c71a7a-8287-4207-81bf-60be096b96d7" (UID: "d0c71a7a-8287-4207-81bf-60be096b96d7"). InnerVolumeSpecName "kube-api-access-277qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:59.534224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:59.534111 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-277qk\" (UniqueName: \"kubernetes.io/projected/d0c71a7a-8287-4207-81bf-60be096b96d7-kube-api-access-277qk\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:13:59.534224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:13:59.534146 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c71a7a-8287-4207-81bf-60be096b96d7-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:00.261489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:00.261452 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" Oct 30 18:14:00.261489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:00.261480 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl" event={"ID":"d0c71a7a-8287-4207-81bf-60be096b96d7","Type":"ContainerDied","Data":"b368afd7154c8ca7e6d6aef2aec5ff7bc81ae8b2c93ec251e1c0984410da2b52"} Oct 30 18:14:00.261751 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:00.261529 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b368afd7154c8ca7e6d6aef2aec5ff7bc81ae8b2c93ec251e1c0984410da2b52" Oct 30 18:14:05.081045 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:14:05.080926 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:14:10.997803 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:10.997761 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h"] Oct 30 18:14:11.000149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:10.998330 4883 cpu_manager.go:414] "containerMap: removing container" podUID="d0c71a7a-8287-4207-81bf-60be096b96d7" containerName="revoke-keys" Oct 30 18:14:11.000149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:10.998344 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c71a7a-8287-4207-81bf-60be096b96d7" containerName="revoke-keys" Oct 30 18:14:11.000149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:10.998465 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c71a7a-8287-4207-81bf-60be096b96d7" containerName="revoke-keys" Oct 30 18:14:11.001855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.001836 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:11.019380 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.019347 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:14:11.019592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.019393 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:14:11.032864 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.032831 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:14:11.032864 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.032835 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:14:11.037669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.037582 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h"] Oct 30 18:14:11.120147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.120097 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2lmf\" (UniqueName: \"kubernetes.io/projected/daa9f1c0-5555-458a-996f-20e18e09c37e-kube-api-access-t2lmf\") pod \"maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h\" (UID: \"daa9f1c0-5555-458a-996f-20e18e09c37e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:11.120403 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.120286 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h\" (UID: \"daa9f1c0-5555-458a-996f-20e18e09c37e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:11.120403 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.120357 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h\" (UID: \"daa9f1c0-5555-458a-996f-20e18e09c37e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:11.221730 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.221686 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2lmf\" (UniqueName: \"kubernetes.io/projected/daa9f1c0-5555-458a-996f-20e18e09c37e-kube-api-access-t2lmf\") pod \"maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h\" (UID: \"daa9f1c0-5555-458a-996f-20e18e09c37e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:11.221955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.221824 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h\" (UID: \"daa9f1c0-5555-458a-996f-20e18e09c37e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:11.221955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.221889 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h\" (UID: \"daa9f1c0-5555-458a-996f-20e18e09c37e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:11.222681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.222650 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h\" (UID: \"daa9f1c0-5555-458a-996f-20e18e09c37e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:11.225152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.225123 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h\" (UID: \"daa9f1c0-5555-458a-996f-20e18e09c37e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:11.247232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.247194 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2lmf\" (UniqueName: \"kubernetes.io/projected/daa9f1c0-5555-458a-996f-20e18e09c37e-kube-api-access-t2lmf\") pod \"maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h\" (UID: \"daa9f1c0-5555-458a-996f-20e18e09c37e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:11.316829 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.316727 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:11.625910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:11.625878 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h"] Oct 30 18:14:11.626762 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:14:11.626729 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa9f1c0_5555_458a_996f_20e18e09c37e.slice/crio-5c28afbbf3cc96983ea5e322860407851ad7194eef0b46d84fb06dac3cd44529 WatchSource:0}: Error finding container 5c28afbbf3cc96983ea5e322860407851ad7194eef0b46d84fb06dac3cd44529: Status 404 returned error can't find the container with id 5c28afbbf3cc96983ea5e322860407851ad7194eef0b46d84fb06dac3cd44529 Oct 30 18:14:12.391582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:12.391541 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" event={"ID":"daa9f1c0-5555-458a-996f-20e18e09c37e","Type":"ContainerStarted","Data":"964e7dc20e5ab39956a29f3515db9381dfab447f8812e0dd1e608dab6191e662"} Oct 30 18:14:12.392047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:12.391608 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" event={"ID":"daa9f1c0-5555-458a-996f-20e18e09c37e","Type":"ContainerStarted","Data":"5c28afbbf3cc96983ea5e322860407851ad7194eef0b46d84fb06dac3cd44529"} Oct 30 18:14:12.437083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:12.437025 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" podStartSLOduration=2.43700977 podStartE2EDuration="2.43700977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:12.434779628 +0000 UTC m=+1311.391783464" watchObservedRunningTime="2026-10-30 18:14:12.43700977 +0000 UTC m=+1311.394013605" Oct 30 18:14:13.401303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:13.401266 4883 generic.go:356] "Generic (PLEG): container finished" podID="daa9f1c0-5555-458a-996f-20e18e09c37e" containerID="964e7dc20e5ab39956a29f3515db9381dfab447f8812e0dd1e608dab6191e662" exitCode=0 Oct 30 18:14:13.401776 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:13.401357 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" event={"ID":"daa9f1c0-5555-458a-996f-20e18e09c37e","Type":"ContainerDied","Data":"964e7dc20e5ab39956a29f3515db9381dfab447f8812e0dd1e608dab6191e662"} Oct 30 18:14:14.570797 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:14.570764 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:14.677333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:14.677240 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-xks-ca\") pod \"daa9f1c0-5555-458a-996f-20e18e09c37e\" (UID: \"daa9f1c0-5555-458a-996f-20e18e09c37e\") " Oct 30 18:14:14.677498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:14.677339 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-openshift-ca\") pod \"daa9f1c0-5555-458a-996f-20e18e09c37e\" (UID: \"daa9f1c0-5555-458a-996f-20e18e09c37e\") " Oct 30 18:14:14.677498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:14.677402 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa9f1c0-5555-458a-996f-20e18e09c37e-kube-api-access-t2lmf\" (UniqueName: \"kubernetes.io/projected/daa9f1c0-5555-458a-996f-20e18e09c37e-kube-api-access-t2lmf\") pod \"daa9f1c0-5555-458a-996f-20e18e09c37e\" (UID: \"daa9f1c0-5555-458a-996f-20e18e09c37e\") " Oct 30 18:14:14.677662 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:14.677637 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-openshift-ca" pod "daa9f1c0-5555-458a-996f-20e18e09c37e" (UID: "daa9f1c0-5555-458a-996f-20e18e09c37e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:14.677811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:14.677791 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:14.679530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:14.679498 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-xks-ca" pod "daa9f1c0-5555-458a-996f-20e18e09c37e" (UID: "daa9f1c0-5555-458a-996f-20e18e09c37e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:14.679625 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:14.679526 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa9f1c0-5555-458a-996f-20e18e09c37e-kube-api-access-t2lmf" pod "daa9f1c0-5555-458a-996f-20e18e09c37e" (UID: "daa9f1c0-5555-458a-996f-20e18e09c37e"). InnerVolumeSpecName "kube-api-access-t2lmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:14.778847 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:14.778803 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa9f1c0-5555-458a-996f-20e18e09c37e-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:14.778847 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:14.778841 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2lmf\" (UniqueName: \"kubernetes.io/projected/daa9f1c0-5555-458a-996f-20e18e09c37e-kube-api-access-t2lmf\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:15.417776 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:15.417746 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" Oct 30 18:14:15.417973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:15.417782 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h" event={"ID":"daa9f1c0-5555-458a-996f-20e18e09c37e","Type":"ContainerDied","Data":"5c28afbbf3cc96983ea5e322860407851ad7194eef0b46d84fb06dac3cd44529"} Oct 30 18:14:15.417973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:15.417841 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c28afbbf3cc96983ea5e322860407851ad7194eef0b46d84fb06dac3cd44529" Oct 30 18:14:21.596528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:21.596494 4883 scope.go:122] "RemoveContainer" containerID="d9d9d0644b4badc85d465ce3b30fcb1233d84a0fa737fc04ea147b32c6aab198" Oct 30 18:14:35.108971 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:14:35.108862 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:14:35.248922 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:35.248862 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl"] Oct 30 18:14:35.261106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:35.261069 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c5891f-vnssl"] Oct 30 18:14:37.181685 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:37.181647 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c71a7a-8287-4207-81bf-60be096b96d7" path="/var/lib/kubelet/pods/d0c71a7a-8287-4207-81bf-60be096b96d7/volumes" Oct 30 18:14:39.268956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:39.262081 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.403374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:39.403329 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:14:39.433103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:39.433041 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.453192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:39.453145 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.538977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:39.538853 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.666288 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:39.666232 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.668359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:39.668319 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.774065 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:39.753435 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.980303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:39.980038 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.087913 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.087859 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.131144 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.131097 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.231188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.231075 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.290729 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.290670 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:14:40.295134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.295079 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.320102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.308036 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:14:40.321064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.321040 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:14:40.473963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.424058 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:14:40.473963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.424975 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:14:40.473963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.427040 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:14:40.474465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.474417 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.474992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.474948 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:14:40.475474 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.475454 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:14:40.475649 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.475635 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:14:40.527206 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.527059 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.598961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.594056 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 18:14:40.598961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.594968 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.624290 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.624243 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:14:40.728323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.728263 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.856543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.801036 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:14:40.857483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.857454 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:14:40.864142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:40.864104 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:14:41.164058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.135792 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:14:41.416239 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.416145 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:14:41.428073 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.223808 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.440205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.261227 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.448990 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.441819 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:14:41.510277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.479088 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.510277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.341353 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:14:41.510277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.203254 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:41.510277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.479848 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.510277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.458099 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:14:41.510277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.316256 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.510277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.458137 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:14:41.510277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.500026 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:14:41.520998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.341314 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:14:41.525266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.525235 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:14:41.525600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.525584 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:14:41.525796 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.525773 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:14:41.525970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.525957 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:14:41.680040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.675393 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:14:41.692145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.692085 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.892015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:41.867153 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:42.290894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.290863 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:14:42.425784 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.425732 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:14:42.517031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.507278 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:14:42.517031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.508348 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:14:42.517031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.508663 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:14:42.517031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.509828 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:14:42.517031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.510400 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:42.517031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.510466 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:14:42.517031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.514900 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:14:42.527387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.527309 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:14:42.659196 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.659153 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:14:42.687979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.671269 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:42.692720 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.692672 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:42.726237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.726207 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:14:42.906993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:42.892728 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.145754 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.144530 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:14:43.158187 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.157258 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:14:43.319959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.316046 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.445160 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.437085 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.528960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.524379 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.528960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.524684 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:14:43.541106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.530680 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.677969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.663638 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:14:43.695831 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.563927 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:43.696142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.696098 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.722487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.721042 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w"] Oct 30 18:14:43.722487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.721337 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" podUID="7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" containerName="istio-proxy" containerID="cri-o://b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30" gracePeriod=30 Oct 30 18:14:43.863723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.734182 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.863723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.740645 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:14:43.863723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.742802 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:14:43.942204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:43.929227 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.139958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.138577 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.229663 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.229625 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq"] Oct 30 18:14:44.240961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.230766 4883 cpu_manager.go:414] "containerMap: removing container" podUID="daa9f1c0-5555-458a-996f-20e18e09c37e" containerName="revoke-keys" Oct 30 18:14:44.241247 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.241213 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa9f1c0-5555-458a-996f-20e18e09c37e" containerName="revoke-keys" Oct 30 18:14:44.241622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.241599 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa9f1c0-5555-458a-996f-20e18e09c37e" containerName="revoke-keys" Oct 30 18:14:44.330228 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.330181 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.402021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.375665 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:14:44.447961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.378062 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:44.464815 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.464768 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.494039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.482232 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:14:44.494039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.482333 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:14:44.494039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.482470 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:14:44.523841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.497012 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq"] Oct 30 18:14:44.523841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.507397 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.523841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.518979 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7np\" (UniqueName: \"kubernetes.io/projected/9eb68642-421e-46c9-9d34-0e60f5694978-kube-api-access-fx7np\") pod \"maas-api-revoke-keys-e2e-api-b-0498d2-twgmq\" (UID: \"9eb68642-421e-46c9-9d34-0e60f5694978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:44.523841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.519059 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0498d2-twgmq\" (UID: \"9eb68642-421e-46c9-9d34-0e60f5694978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:44.523841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.519255 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0498d2-twgmq\" (UID: \"9eb68642-421e-46c9-9d34-0e60f5694978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:44.524801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.524760 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:14:44.525701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.525667 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:14:44.537086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.537026 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.555515 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.555467 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.616513 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.581283 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:14:44.761017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.632839 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0498d2-twgmq\" (UID: \"9eb68642-421e-46c9-9d34-0e60f5694978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:44.854961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.854222 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7np\" (UniqueName: \"kubernetes.io/projected/9eb68642-421e-46c9-9d34-0e60f5694978-kube-api-access-fx7np\") pod \"maas-api-revoke-keys-e2e-api-b-0498d2-twgmq\" (UID: \"9eb68642-421e-46c9-9d34-0e60f5694978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:44.854961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.854317 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0498d2-twgmq\" (UID: \"9eb68642-421e-46c9-9d34-0e60f5694978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:44.854961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.827809 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:14:44.854961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.764409 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0498d2-twgmq\" (UID: \"9eb68642-421e-46c9-9d34-0e60f5694978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:44.854961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.828311 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.877972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.764485 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.877972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.828269 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:14:44.948138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:44.948076 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.000178 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.000141 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0498d2-twgmq\" (UID: \"9eb68642-421e-46c9-9d34-0e60f5694978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:45.028419 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.028281 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7np\" (UniqueName: \"kubernetes.io/projected/9eb68642-421e-46c9-9d34-0e60f5694978-kube-api-access-fx7np\") pod \"maas-api-revoke-keys-e2e-api-b-0498d2-twgmq\" (UID: \"9eb68642-421e-46c9-9d34-0e60f5694978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:45.173186 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.139640 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.295273 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.295171 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:45.344960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.331030 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.472959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.454344 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:14:45.476416 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.475392 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.547022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.532126 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.547022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.532595 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:14:45.547022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.533042 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:14:45.547022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.537467 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.550885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.550821 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.573569 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.573534 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:14:45.598317 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.598274 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.604964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.604877 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.863701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.863658 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:14:45.907015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.889709 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:14:45.960333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.960282 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:45.988450 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.960673 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:46.011121 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:45.993729 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:14:46.091962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.059017 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:14:46.145444 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.144128 4883 patch_prober.go:29] interesting pod/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:14:46.145444 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.144197 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" podUID="7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:14:46.145444 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.064083 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:46.251428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.251370 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:46.367079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.367029 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:46.484186 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.484068 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:46.536159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.536086 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:14:46.537854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.537804 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:14:46.544255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.544195 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:14:46.594008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.593962 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:46.716979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.611556 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:46.894842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.894079 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:46.894842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.894384 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:14:46.938291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.928217 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:14:46.938291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.928280 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:14:46.995358 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.989961 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.019113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:46.997995 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.108986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.105019 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:14:47.119041 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.111966 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:14:47.161145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.161035 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.176960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.164603 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq"] Oct 30 18:14:47.241217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.225693 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:14:47.241217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.225762 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:14:47.250194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.249013 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:14:47.258958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.255777 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.285361 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.284816 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:14:47.298723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.298661 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:14:47.327077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.327048 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:14:47.338589 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.338557 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:14:47.341059 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.341027 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:14:47.341757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.341730 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:14:47.359671 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.359244 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:14:47.981499 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.981395 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" event={"ID":"9eb68642-421e-46c9-9d34-0e60f5694978","Type":"ContainerStarted","Data":"11289482ef4c3d863f26f03703f14c2eef1f1373b520a1dd9dbad5561453c8d6"} Oct 30 18:14:47.982006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:47.981501 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" event={"ID":"9eb68642-421e-46c9-9d34-0e60f5694978","Type":"ContainerStarted","Data":"18558544bb31f658010a77567ec2716c93169b297f0a6da4a4f1d837adb15e47"} Oct 30 18:14:48.028896 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:48.028829 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" podStartSLOduration=5.028813184 podStartE2EDuration="5.028813184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:48.024487497 +0000 UTC m=+1346.981491334" watchObservedRunningTime="2026-10-30 18:14:48.028813184 +0000 UTC m=+1346.985817020" Oct 30 18:14:48.999958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:48.999648 4883 generic.go:356] "Generic (PLEG): container finished" podID="9eb68642-421e-46c9-9d34-0e60f5694978" containerID="11289482ef4c3d863f26f03703f14c2eef1f1373b520a1dd9dbad5561453c8d6" exitCode=0 Oct 30 18:14:48.999958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:48.999783 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" event={"ID":"9eb68642-421e-46c9-9d34-0e60f5694978","Type":"ContainerDied","Data":"11289482ef4c3d863f26f03703f14c2eef1f1373b520a1dd9dbad5561453c8d6"} Oct 30 18:14:49.450043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.450010 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.543013 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-certs\") pod \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.543131 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-envoy\") pod \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.543169 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-kube-api-access-f4dp6\" (UniqueName: \"kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-kube-api-access-f4dp6\") pod \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.543322 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-data\") pod \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.543337 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-certs" pod "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" (UID: "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.543362 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-token\" (UniqueName: \"kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-token\") pod \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.543474 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-podinfo\") pod \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.543505 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-credential-socket\") pod \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.543564 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-socket\") pod \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.543657 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istiod-ca-cert\") pod \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\" (UID: \"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb\") " Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.544325 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.544820 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-socket" pod "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" (UID: "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.544912 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-data" pod "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" (UID: "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:14:49.545091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.544998 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-credential-socket" pod "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" (UID: "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:14:49.546166 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.545149 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istiod-ca-cert" pod "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" (UID: "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:49.547785 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.547382 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-kube-api-access-f4dp6" pod "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" (UID: "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb"). InnerVolumeSpecName "kube-api-access-f4dp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:49.549328 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.549293 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-podinfo" pod "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" (UID: "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:14:49.551048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.551000 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-token" pod "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" (UID: "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:49.551864 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.551837 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-envoy" pod "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" (UID: "7cd4bf78-d814-4cd9-80b2-730ae3eebaeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:14:49.645639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.645584 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:49.645639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.645624 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:49.645639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.645633 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:49.645639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.645642 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:49.645639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.645651 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:49.646077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.645662 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:49.646077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.645674 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:49.646077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:49.645685 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4dp6\" (UniqueName: \"kubernetes.io/projected/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb-kube-api-access-f4dp6\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:50.010656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.010558 4883 generic.go:356] "Generic (PLEG): container finished" podID="7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" containerID="b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30" exitCode=0 Oct 30 18:14:50.010656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.010622 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" Oct 30 18:14:50.011251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.010648 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" event={"ID":"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb","Type":"ContainerDied","Data":"b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30"} Oct 30 18:14:50.011251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.010718 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w" event={"ID":"7cd4bf78-d814-4cd9-80b2-730ae3eebaeb","Type":"ContainerDied","Data":"89b0e3298e456d33e356f445b7e093edecca9c2c16bf63d3a6650c2c3257b8d7"} Oct 30 18:14:50.011251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.010750 4883 scope.go:122] "RemoveContainer" containerID="b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30" Oct 30 18:14:50.023620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.023586 4883 scope.go:122] "RemoveContainer" containerID="b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30" Oct 30 18:14:50.024121 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:14:50.024098 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30\": container with ID starting with b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30 not found: ID does not exist" containerID="b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30" Oct 30 18:14:50.024212 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.024132 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30"} err="failed to get container status \"b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30\": rpc error: code = NotFound desc = could not find container \"b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30\": container with ID starting with b78d00e0289f2bce37de162885816c675353feb669642a2124de8dfd694eaf30 not found: ID does not exist" Oct 30 18:14:50.069533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.069492 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w"] Oct 30 18:14:50.093366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.093325 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c5891f-openshift-default-df5849fc7-bqh7w"] Oct 30 18:14:50.195156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.195120 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:50.355022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.354981 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-xks-ca\") pod \"9eb68642-421e-46c9-9d34-0e60f5694978\" (UID: \"9eb68642-421e-46c9-9d34-0e60f5694978\") " Oct 30 18:14:50.355188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.355122 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb68642-421e-46c9-9d34-0e60f5694978-kube-api-access-fx7np\" (UniqueName: \"kubernetes.io/projected/9eb68642-421e-46c9-9d34-0e60f5694978-kube-api-access-fx7np\") pod \"9eb68642-421e-46c9-9d34-0e60f5694978\" (UID: \"9eb68642-421e-46c9-9d34-0e60f5694978\") " Oct 30 18:14:50.355188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.355158 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-openshift-ca\") pod \"9eb68642-421e-46c9-9d34-0e60f5694978\" (UID: \"9eb68642-421e-46c9-9d34-0e60f5694978\") " Oct 30 18:14:50.355599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.355564 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-openshift-ca" pod "9eb68642-421e-46c9-9d34-0e60f5694978" (UID: "9eb68642-421e-46c9-9d34-0e60f5694978"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:50.357444 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.357417 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-xks-ca" pod "9eb68642-421e-46c9-9d34-0e60f5694978" (UID: "9eb68642-421e-46c9-9d34-0e60f5694978"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:50.357444 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.357433 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb68642-421e-46c9-9d34-0e60f5694978-kube-api-access-fx7np" pod "9eb68642-421e-46c9-9d34-0e60f5694978" (UID: "9eb68642-421e-46c9-9d34-0e60f5694978"). InnerVolumeSpecName "kube-api-access-fx7np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:50.456530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.456479 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx7np\" (UniqueName: \"kubernetes.io/projected/9eb68642-421e-46c9-9d34-0e60f5694978-kube-api-access-fx7np\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:50.456530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.456518 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:50.456530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:50.456533 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb68642-421e-46c9-9d34-0e60f5694978-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:14:51.021572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:51.021538 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" Oct 30 18:14:51.022074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:51.021549 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq" event={"ID":"9eb68642-421e-46c9-9d34-0e60f5694978","Type":"ContainerDied","Data":"18558544bb31f658010a77567ec2716c93169b297f0a6da4a4f1d837adb15e47"} Oct 30 18:14:51.022074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:51.021692 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18558544bb31f658010a77567ec2716c93169b297f0a6da4a4f1d837adb15e47" Oct 30 18:14:51.180889 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:51.180846 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" path="/var/lib/kubelet/pods/7cd4bf78-d814-4cd9-80b2-730ae3eebaeb/volumes" Oct 30 18:14:53.036345 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:53.036305 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz"] Oct 30 18:14:53.048999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:53.048923 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz"] Oct 30 18:14:53.055258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:53.055204 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6043a3-6qdfz"] Oct 30 18:14:53.061108 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:53.061062 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee44a3b-x28hz"] Oct 30 18:14:53.179903 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:53.179863 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32bf2b1f-efc8-4b29-87df-55d5606d3e46" path="/var/lib/kubelet/pods/32bf2b1f-efc8-4b29-87df-55d5606d3e46/volumes" Oct 30 18:14:53.180421 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:53.180402 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d89d01-b717-4336-9951-620ef9dce005" path="/var/lib/kubelet/pods/b7d89d01-b717-4336-9951-620ef9dce005/volumes" Oct 30 18:14:55.218482 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.218441 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s"] Oct 30 18:14:55.219332 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.219289 4883 cpu_manager.go:414] "containerMap: removing container" podUID="9eb68642-421e-46c9-9d34-0e60f5694978" containerName="revoke-keys" Oct 30 18:14:55.219332 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.219331 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb68642-421e-46c9-9d34-0e60f5694978" containerName="revoke-keys" Oct 30 18:14:55.219514 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.219349 4883 cpu_manager.go:414] "containerMap: removing container" podUID="7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" containerName="istio-proxy" Oct 30 18:14:55.219514 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.219356 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" containerName="istio-proxy" Oct 30 18:14:55.219613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.219559 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd4bf78-d814-4cd9-80b2-730ae3eebaeb" containerName="istio-proxy" Oct 30 18:14:55.219613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.219576 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb68642-421e-46c9-9d34-0e60f5694978" containerName="revoke-keys" Oct 30 18:14:55.225074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.225042 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.254813 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.254780 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0bdf03-openshift-default-dockercfg-krrwb\"" Oct 30 18:14:55.262227 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.262189 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s"] Oct 30 18:14:55.341955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.336286 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-token\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.341955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.336414 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-credential-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.341955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.336466 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-envoy\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.341955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.336551 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-data\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.341955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.336600 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-podinfo\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.341955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.336696 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.341955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.336757 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-certs\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.341955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.336807 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88c92\" (UniqueName: \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-kube-api-access-88c92\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.341955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.336840 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istiod-ca-cert\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.438161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.438118 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-token\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.438376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.438235 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-credential-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.438376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.438277 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-envoy\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.438376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.438350 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-data\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.438570 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.438399 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-podinfo\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.438570 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.438428 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.438570 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.438495 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-certs\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.438570 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.438547 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88c92\" (UniqueName: \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-kube-api-access-88c92\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.438765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.438580 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istiod-ca-cert\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.438765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.438750 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-credential-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.438868 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.438837 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-data\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.439057 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.439033 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-socket\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.439389 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.439368 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istiod-ca-cert\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.439491 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.439430 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-certs\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.441453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.441428 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-envoy\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.441988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.441843 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-podinfo\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.463694 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.463647 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88c92\" (UniqueName: \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-kube-api-access-88c92\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.464049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.464015 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-token\") pod \"e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.543243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.543147 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:55.882089 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:14:55.882048 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269010f9_cf3f_48ee_ba1d_8fdb5404a937.slice/crio-e561b5ba6b0da55a1e0b0eed17157731d42b0c9cd2c89022110e2899e809bdf7 WatchSource:0}: Error finding container e561b5ba6b0da55a1e0b0eed17157731d42b0c9cd2c89022110e2899e809bdf7: Status 404 returned error can't find the container with id e561b5ba6b0da55a1e0b0eed17157731d42b0c9cd2c89022110e2899e809bdf7 Oct 30 18:14:55.882599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.882570 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s"] Oct 30 18:14:55.884615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.884579 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:14:55.884700 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.884648 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:14:55.884700 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:55.884679 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:14:56.076999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:56.076924 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" event={"ID":"269010f9-cf3f-48ee-ba1d-8fdb5404a937","Type":"ContainerStarted","Data":"290c6642f292a3a3d5f989ca87da6d6b240d2aaf0413d6f8dfa7b56d78d6c9e9"} Oct 30 18:14:56.077176 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:56.077017 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" event={"ID":"269010f9-cf3f-48ee-ba1d-8fdb5404a937","Type":"ContainerStarted","Data":"e561b5ba6b0da55a1e0b0eed17157731d42b0c9cd2c89022110e2899e809bdf7"} Oct 30 18:14:56.130533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:56.130373 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" podStartSLOduration=1.130349847 podStartE2EDuration="1.130349847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:56.12353095 +0000 UTC m=+1355.080534793" watchObservedRunningTime="2026-10-30 18:14:56.130349847 +0000 UTC m=+1355.087353684" Oct 30 18:14:56.560071 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:56.546101 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:14:56.560071 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:56.557465 4883 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:14:56.560071 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:56.557527 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" podUID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:14:57.013322 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:57.013273 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h"] Oct 30 18:14:57.019885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:57.019842 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c8e1a-87k7h"] Oct 30 18:14:57.203458 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:57.203414 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa9f1c0-5555-458a-996f-20e18e09c37e" path="/var/lib/kubelet/pods/daa9f1c0-5555-458a-996f-20e18e09c37e/volumes" Oct 30 18:14:57.544045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:57.544003 4883 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:14:57.544275 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:57.544067 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" podUID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:14:58.543987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:58.543924 4883 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:14:58.544393 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:58.544008 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" podUID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:14:59.543565 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:59.543525 4883 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:14:59.543791 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:14:59.543596 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" podUID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:15:00.144269 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.144235 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57"] Oct 30 18:15:00.149920 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.149888 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw"] Oct 30 18:15:00.150146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.150122 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" Oct 30 18:15:00.154965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.154921 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds"] Oct 30 18:15:00.155134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.155027 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" Oct 30 18:15:00.159371 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.159340 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8"] Oct 30 18:15:00.159668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.159639 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" Oct 30 18:15:00.163802 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.163778 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" Oct 30 18:15:00.192768 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.182366 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:15:00.192768 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.183280 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:15:00.199217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.199176 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:15:00.204371 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.203383 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp"] Oct 30 18:15:00.216344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.216311 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57"] Oct 30 18:15:00.216521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.216362 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw"] Oct 30 18:15:00.216521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.216461 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" Oct 30 18:15:00.223766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.223735 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds"] Oct 30 18:15:00.228076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.228040 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8"] Oct 30 18:15:00.242828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.242793 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp"] Oct 30 18:15:00.250788 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.250747 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1021fc1b-74a1-49ae-865c-5d357bfb2c25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57\" (UID: \"1021fc1b-74a1-49ae-865c-5d357bfb2c25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" Oct 30 18:15:00.251000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.250809 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzncj\" (UniqueName: \"kubernetes.io/projected/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-kube-api-access-nzncj\") pod \"maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw\" (UID: \"cd6fb8b2-f083-490e-8ed7-c96a99f01b22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" Oct 30 18:15:00.251000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.250978 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnr7\" (UniqueName: \"kubernetes.io/projected/1021fc1b-74a1-49ae-865c-5d357bfb2c25-kube-api-access-gsnr7\") pod \"maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57\" (UID: \"1021fc1b-74a1-49ae-865c-5d357bfb2c25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" Oct 30 18:15:00.251142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.251109 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw\" (UID: \"cd6fb8b2-f083-490e-8ed7-c96a99f01b22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" Oct 30 18:15:00.251201 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.251151 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szdxp\" (UniqueName: \"kubernetes.io/projected/3c689fe2-b26b-49f0-b37b-902238098837-kube-api-access-szdxp\") pod \"maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds\" (UID: \"3c689fe2-b26b-49f0-b37b-902238098837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" Oct 30 18:15:00.251590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.251546 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e786833-606f-4c49-9d09-56bef62b1a4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8\" (UID: \"7e786833-606f-4c49-9d09-56bef62b1a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" Oct 30 18:15:00.251709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.251620 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229ts\" (UniqueName: \"kubernetes.io/projected/7e786833-606f-4c49-9d09-56bef62b1a4a-kube-api-access-229ts\") pod \"maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8\" (UID: \"7e786833-606f-4c49-9d09-56bef62b1a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" Oct 30 18:15:00.251709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.251654 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c689fe2-b26b-49f0-b37b-902238098837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds\" (UID: \"3c689fe2-b26b-49f0-b37b-902238098837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" Oct 30 18:15:00.353364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.353328 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp\" (UID: \"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" Oct 30 18:15:00.353586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.353493 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e786833-606f-4c49-9d09-56bef62b1a4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8\" (UID: \"7e786833-606f-4c49-9d09-56bef62b1a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" Oct 30 18:15:00.353586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.353546 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bfjz\" (UniqueName: \"kubernetes.io/projected/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-kube-api-access-2bfjz\") pod \"maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp\" (UID: \"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" Oct 30 18:15:00.353685 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.353609 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229ts\" (UniqueName: \"kubernetes.io/projected/7e786833-606f-4c49-9d09-56bef62b1a4a-kube-api-access-229ts\") pod \"maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8\" (UID: \"7e786833-606f-4c49-9d09-56bef62b1a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" Oct 30 18:15:00.353685 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.353660 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c689fe2-b26b-49f0-b37b-902238098837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds\" (UID: \"3c689fe2-b26b-49f0-b37b-902238098837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" Oct 30 18:15:00.353770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.353732 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1021fc1b-74a1-49ae-865c-5d357bfb2c25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57\" (UID: \"1021fc1b-74a1-49ae-865c-5d357bfb2c25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" Oct 30 18:15:00.353823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.353771 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzncj\" (UniqueName: \"kubernetes.io/projected/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-kube-api-access-nzncj\") pod \"maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw\" (UID: \"cd6fb8b2-f083-490e-8ed7-c96a99f01b22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" Oct 30 18:15:00.353823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.353814 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnr7\" (UniqueName: \"kubernetes.io/projected/1021fc1b-74a1-49ae-865c-5d357bfb2c25-kube-api-access-gsnr7\") pod \"maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57\" (UID: \"1021fc1b-74a1-49ae-865c-5d357bfb2c25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" Oct 30 18:15:00.353924 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.353909 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw\" (UID: \"cd6fb8b2-f083-490e-8ed7-c96a99f01b22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" Oct 30 18:15:00.354008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.353960 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szdxp\" (UniqueName: \"kubernetes.io/projected/3c689fe2-b26b-49f0-b37b-902238098837-kube-api-access-szdxp\") pod \"maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds\" (UID: \"3c689fe2-b26b-49f0-b37b-902238098837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" Oct 30 18:15:00.354272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.354248 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e786833-606f-4c49-9d09-56bef62b1a4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8\" (UID: \"7e786833-606f-4c49-9d09-56bef62b1a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" Oct 30 18:15:00.354502 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.354482 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c689fe2-b26b-49f0-b37b-902238098837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds\" (UID: \"3c689fe2-b26b-49f0-b37b-902238098837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" Oct 30 18:15:00.354563 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.354510 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1021fc1b-74a1-49ae-865c-5d357bfb2c25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57\" (UID: \"1021fc1b-74a1-49ae-865c-5d357bfb2c25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" Oct 30 18:15:00.354644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.354618 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw\" (UID: \"cd6fb8b2-f083-490e-8ed7-c96a99f01b22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" Oct 30 18:15:00.404100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.404006 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-229ts\" (UniqueName: \"kubernetes.io/projected/7e786833-606f-4c49-9d09-56bef62b1a4a-kube-api-access-229ts\") pod \"maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8\" (UID: \"7e786833-606f-4c49-9d09-56bef62b1a4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" Oct 30 18:15:00.405725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.405099 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnr7\" (UniqueName: \"kubernetes.io/projected/1021fc1b-74a1-49ae-865c-5d357bfb2c25-kube-api-access-gsnr7\") pod \"maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57\" (UID: \"1021fc1b-74a1-49ae-865c-5d357bfb2c25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" Oct 30 18:15:00.405725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.405147 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szdxp\" (UniqueName: \"kubernetes.io/projected/3c689fe2-b26b-49f0-b37b-902238098837-kube-api-access-szdxp\") pod \"maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds\" (UID: \"3c689fe2-b26b-49f0-b37b-902238098837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" Oct 30 18:15:00.405725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.405494 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzncj\" (UniqueName: \"kubernetes.io/projected/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-kube-api-access-nzncj\") pod \"maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw\" (UID: \"cd6fb8b2-f083-490e-8ed7-c96a99f01b22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" Oct 30 18:15:00.455252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.455208 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp\" (UID: \"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" Oct 30 18:15:00.455437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.455360 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfjz\" (UniqueName: \"kubernetes.io/projected/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-kube-api-access-2bfjz\") pod \"maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp\" (UID: \"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" Oct 30 18:15:00.456014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.455990 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp\" (UID: \"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" Oct 30 18:15:00.501768 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.501734 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bfjz\" (UniqueName: \"kubernetes.io/projected/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-kube-api-access-2bfjz\") pod \"maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp\" (UID: \"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" Oct 30 18:15:00.502033 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.502010 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" Oct 30 18:15:00.526132 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.526088 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" Oct 30 18:15:00.534618 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.534584 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" Oct 30 18:15:00.543548 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.543503 4883 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:15:00.543691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.543612 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" podUID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:15:00.549688 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.549653 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" Oct 30 18:15:00.602311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.602264 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" Oct 30 18:15:00.855275 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:00.855240 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57"] Oct 30 18:15:00.855704 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:15:00.855677 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1021fc1b_74a1_49ae_865c_5d357bfb2c25.slice/crio-cce94e7c4283b22c69fbca1d80cc4d222dd8b5078607ac5075119a4f158bdb61 WatchSource:0}: Error finding container cce94e7c4283b22c69fbca1d80cc4d222dd8b5078607ac5075119a4f158bdb61: Status 404 returned error can't find the container with id cce94e7c4283b22c69fbca1d80cc4d222dd8b5078607ac5075119a4f158bdb61 Oct 30 18:15:01.011747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:01.010994 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp"] Oct 30 18:15:01.097232 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:15:01.097177 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e99ea40_c6f2_4bb2_a2c1_d62304538ec3.slice/crio-7373ff85da40a1e79de689dda8ef016188b59108c2f8984b3e3edd6e2fed0cad WatchSource:0}: Error finding container 7373ff85da40a1e79de689dda8ef016188b59108c2f8984b3e3edd6e2fed0cad: Status 404 returned error can't find the container with id 7373ff85da40a1e79de689dda8ef016188b59108c2f8984b3e3edd6e2fed0cad Oct 30 18:15:01.132725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:01.132686 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" event={"ID":"1021fc1b-74a1-49ae-865c-5d357bfb2c25","Type":"ContainerStarted","Data":"cce94e7c4283b22c69fbca1d80cc4d222dd8b5078607ac5075119a4f158bdb61"} Oct 30 18:15:01.134230 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:01.134182 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" event={"ID":"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3","Type":"ContainerStarted","Data":"7373ff85da40a1e79de689dda8ef016188b59108c2f8984b3e3edd6e2fed0cad"} Oct 30 18:15:01.225355 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:01.225220 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds"] Oct 30 18:15:01.225355 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:01.225290 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8"] Oct 30 18:15:01.230928 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:01.230870 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw"] Oct 30 18:15:01.544583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:01.544542 4883 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:15:01.544766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:01.544605 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" podUID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:15:02.143944 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.143886 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" event={"ID":"7e786833-606f-4c49-9d09-56bef62b1a4a","Type":"ContainerStarted","Data":"50462c2f3c2549313cc82f5b668b02d7329f34f4713c1585faf76c051884baa5"} Oct 30 18:15:02.144147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.143992 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" event={"ID":"7e786833-606f-4c49-9d09-56bef62b1a4a","Type":"ContainerStarted","Data":"9ed5f47998200d832e4efc3d1a9506f563eab458be8e576aef2e3d7e6544d5d0"} Oct 30 18:15:02.145897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.145810 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" event={"ID":"3c689fe2-b26b-49f0-b37b-902238098837","Type":"ContainerStarted","Data":"20b5896245bb8d1e4b27319556a9b733add2b58b4c66f7e159ce5554af623781"} Oct 30 18:15:02.145897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.145873 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" event={"ID":"3c689fe2-b26b-49f0-b37b-902238098837","Type":"ContainerStarted","Data":"623c9306bfdaf06ee29b7c7599e8376d22032c3d351bae65be1e2030128d5389"} Oct 30 18:15:02.147617 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.147588 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" event={"ID":"cd6fb8b2-f083-490e-8ed7-c96a99f01b22","Type":"ContainerStarted","Data":"97833565c1d9062233cdfb371a01110f18293d948620b2702aeffc8648ded139"} Oct 30 18:15:02.147766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.147640 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" event={"ID":"cd6fb8b2-f083-490e-8ed7-c96a99f01b22","Type":"ContainerStarted","Data":"ec4c5eb8118565ece8781fc8af2466c9b7f8e1cdbb7a239f419b7cb9ed7681d0"} Oct 30 18:15:02.152695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.152660 4883 generic.go:356] "Generic (PLEG): container finished" podID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" containerID="4a19d67814c33c833335e1fa9aeedc36b1a1ddeabd8c406cdec136bfe5a6e141" exitCode=60 Oct 30 18:15:02.153167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.153145 4883 scope.go:122] "RemoveContainer" containerID="4a19d67814c33c833335e1fa9aeedc36b1a1ddeabd8c406cdec136bfe5a6e141" Oct 30 18:15:02.153372 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.153333 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" event={"ID":"1021fc1b-74a1-49ae-865c-5d357bfb2c25","Type":"ContainerDied","Data":"4a19d67814c33c833335e1fa9aeedc36b1a1ddeabd8c406cdec136bfe5a6e141"} Oct 30 18:15:02.154787 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.154758 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" event={"ID":"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3","Type":"ContainerStarted","Data":"38b35aefa66aa018079ff6e9d200d023eb7898d183b0fec3cd968a417ec7b2e4"} Oct 30 18:15:02.190846 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.190772 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" podStartSLOduration=2.190747736 podStartE2EDuration="2.190747736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:02.185462988 +0000 UTC m=+1361.142466820" watchObservedRunningTime="2026-10-30 18:15:02.190747736 +0000 UTC m=+1361.147751573" Oct 30 18:15:02.273076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.272891 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" podStartSLOduration=2.272861125 podStartE2EDuration="2.272861125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:02.27141716 +0000 UTC m=+1361.228420997" watchObservedRunningTime="2026-10-30 18:15:02.272861125 +0000 UTC m=+1361.229864961" Oct 30 18:15:02.313579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.313511 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" podStartSLOduration=2.313491676 podStartE2EDuration="2.313491676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:02.312842762 +0000 UTC m=+1361.269846599" watchObservedRunningTime="2026-10-30 18:15:02.313491676 +0000 UTC m=+1361.270495512" Oct 30 18:15:02.359398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.359337 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" podStartSLOduration=2.3593184320000002 podStartE2EDuration="2.359318432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:02.354425951 +0000 UTC m=+1361.311429789" watchObservedRunningTime="2026-10-30 18:15:02.359318432 +0000 UTC m=+1361.316322270" Oct 30 18:15:02.549138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.549089 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:15:02.549541 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.549518 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:15:02.550487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:02.550465 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:15:03.169138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.169092 4883 generic.go:356] "Generic (PLEG): container finished" podID="7e786833-606f-4c49-9d09-56bef62b1a4a" containerID="50462c2f3c2549313cc82f5b668b02d7329f34f4713c1585faf76c051884baa5" exitCode=60 Oct 30 18:15:03.169319 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.169173 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" event={"ID":"7e786833-606f-4c49-9d09-56bef62b1a4a","Type":"ContainerDied","Data":"50462c2f3c2549313cc82f5b668b02d7329f34f4713c1585faf76c051884baa5"} Oct 30 18:15:03.169525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.169509 4883 scope.go:122] "RemoveContainer" containerID="50462c2f3c2549313cc82f5b668b02d7329f34f4713c1585faf76c051884baa5" Oct 30 18:15:03.171103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.171081 4883 generic.go:356] "Generic (PLEG): container finished" podID="3c689fe2-b26b-49f0-b37b-902238098837" containerID="20b5896245bb8d1e4b27319556a9b733add2b58b4c66f7e159ce5554af623781" exitCode=60 Oct 30 18:15:03.171484 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.171467 4883 scope.go:122] "RemoveContainer" containerID="20b5896245bb8d1e4b27319556a9b733add2b58b4c66f7e159ce5554af623781" Oct 30 18:15:03.173092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.173001 4883 generic.go:356] "Generic (PLEG): container finished" podID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" containerID="97833565c1d9062233cdfb371a01110f18293d948620b2702aeffc8648ded139" exitCode=60 Oct 30 18:15:03.173432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.173346 4883 scope.go:122] "RemoveContainer" containerID="97833565c1d9062233cdfb371a01110f18293d948620b2702aeffc8648ded139" Oct 30 18:15:03.177252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.177229 4883 generic.go:356] "Generic (PLEG): container finished" podID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" containerID="38b35aefa66aa018079ff6e9d200d023eb7898d183b0fec3cd968a417ec7b2e4" exitCode=60 Oct 30 18:15:03.177605 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.177585 4883 scope.go:122] "RemoveContainer" containerID="38b35aefa66aa018079ff6e9d200d023eb7898d183b0fec3cd968a417ec7b2e4" Oct 30 18:15:03.180365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.180336 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" event={"ID":"3c689fe2-b26b-49f0-b37b-902238098837","Type":"ContainerDied","Data":"20b5896245bb8d1e4b27319556a9b733add2b58b4c66f7e159ce5554af623781"} Oct 30 18:15:03.180482 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.180395 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" event={"ID":"cd6fb8b2-f083-490e-8ed7-c96a99f01b22","Type":"ContainerDied","Data":"97833565c1d9062233cdfb371a01110f18293d948620b2702aeffc8648ded139"} Oct 30 18:15:03.180482 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.180423 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" event={"ID":"1021fc1b-74a1-49ae-865c-5d357bfb2c25","Type":"ContainerStarted","Data":"e079f946f6031f9a252b2b9c5598d5b2ef38b1b9c79f884a5d1c0b06e633d4f3"} Oct 30 18:15:03.180482 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.180455 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" event={"ID":"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3","Type":"ContainerDied","Data":"38b35aefa66aa018079ff6e9d200d023eb7898d183b0fec3cd968a417ec7b2e4"} Oct 30 18:15:03.253930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:03.253881 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" podStartSLOduration=3.253865938 podStartE2EDuration="3.253865938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:03.253078908 +0000 UTC m=+1362.210082742" watchObservedRunningTime="2026-10-30 18:15:03.253865938 +0000 UTC m=+1362.210869775" Oct 30 18:15:04.186675 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:04.186638 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" event={"ID":"3c689fe2-b26b-49f0-b37b-902238098837","Type":"ContainerStarted","Data":"f22b43db4e9def85b7feb2078fbd68afb9cb74aeb772c92dfe946c6a1ca1ac22"} Oct 30 18:15:04.189220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:04.189191 4883 generic.go:356] "Generic (PLEG): container finished" podID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" containerID="95368ca64cdf06367c6906a272dae06548d4f904f4ef5b2518c0c4c2d246b8ba" exitCode=60 Oct 30 18:15:04.189368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:04.189273 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" event={"ID":"cd6fb8b2-f083-490e-8ed7-c96a99f01b22","Type":"ContainerDied","Data":"95368ca64cdf06367c6906a272dae06548d4f904f4ef5b2518c0c4c2d246b8ba"} Oct 30 18:15:04.189368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:04.189342 4883 scope.go:122] "RemoveContainer" containerID="97833565c1d9062233cdfb371a01110f18293d948620b2702aeffc8648ded139" Oct 30 18:15:04.189704 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:04.189688 4883 scope.go:122] "RemoveContainer" containerID="95368ca64cdf06367c6906a272dae06548d4f904f4ef5b2518c0c4c2d246b8ba" Oct 30 18:15:04.190072 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:04.190052 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw_odh-ai-gateway-infra(cd6fb8b2-f083-490e-8ed7-c96a99f01b22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" Oct 30 18:15:04.191520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:04.191498 4883 generic.go:356] "Generic (PLEG): container finished" podID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" containerID="e079f946f6031f9a252b2b9c5598d5b2ef38b1b9c79f884a5d1c0b06e633d4f3" exitCode=60 Oct 30 18:15:04.191605 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:04.191585 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" event={"ID":"1021fc1b-74a1-49ae-865c-5d357bfb2c25","Type":"ContainerDied","Data":"e079f946f6031f9a252b2b9c5598d5b2ef38b1b9c79f884a5d1c0b06e633d4f3"} Oct 30 18:15:04.191860 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:04.191844 4883 scope.go:122] "RemoveContainer" containerID="e079f946f6031f9a252b2b9c5598d5b2ef38b1b9c79f884a5d1c0b06e633d4f3" Oct 30 18:15:04.192205 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:04.192187 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57_odh-ai-gateway-infra(1021fc1b-74a1-49ae-865c-5d357bfb2c25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" Oct 30 18:15:04.193786 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:04.193758 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" event={"ID":"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3","Type":"ContainerStarted","Data":"0974b2d4d1daef0519993e990c244c69a5c6a6a21cd83eb85b91bf45e05d34fb"} Oct 30 18:15:04.196688 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:04.196652 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" event={"ID":"7e786833-606f-4c49-9d09-56bef62b1a4a","Type":"ContainerStarted","Data":"fe5adcbf5f7629d9dfabd8b83852286f76da2dbfc775745464b9682caaf0eec4"} Oct 30 18:15:04.242411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:04.242137 4883 scope.go:122] "RemoveContainer" containerID="4a19d67814c33c833335e1fa9aeedc36b1a1ddeabd8c406cdec136bfe5a6e141" Oct 30 18:15:05.082674 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:05.082638 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:15:05.207814 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.207779 4883 generic.go:356] "Generic (PLEG): container finished" podID="7e786833-606f-4c49-9d09-56bef62b1a4a" containerID="fe5adcbf5f7629d9dfabd8b83852286f76da2dbfc775745464b9682caaf0eec4" exitCode=60 Oct 30 18:15:05.208355 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.207868 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" event={"ID":"7e786833-606f-4c49-9d09-56bef62b1a4a","Type":"ContainerDied","Data":"fe5adcbf5f7629d9dfabd8b83852286f76da2dbfc775745464b9682caaf0eec4"} Oct 30 18:15:05.208355 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.207953 4883 scope.go:122] "RemoveContainer" containerID="50462c2f3c2549313cc82f5b668b02d7329f34f4713c1585faf76c051884baa5" Oct 30 18:15:05.208355 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.208212 4883 scope.go:122] "RemoveContainer" containerID="fe5adcbf5f7629d9dfabd8b83852286f76da2dbfc775745464b9682caaf0eec4" Oct 30 18:15:05.208661 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:05.208637 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8_odh-ai-gateway-infra(7e786833-606f-4c49-9d09-56bef62b1a4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" Oct 30 18:15:05.210282 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.210257 4883 generic.go:356] "Generic (PLEG): container finished" podID="3c689fe2-b26b-49f0-b37b-902238098837" containerID="f22b43db4e9def85b7feb2078fbd68afb9cb74aeb772c92dfe946c6a1ca1ac22" exitCode=60 Oct 30 18:15:05.210387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.210336 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" event={"ID":"3c689fe2-b26b-49f0-b37b-902238098837","Type":"ContainerDied","Data":"f22b43db4e9def85b7feb2078fbd68afb9cb74aeb772c92dfe946c6a1ca1ac22"} Oct 30 18:15:05.210653 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.210637 4883 scope.go:122] "RemoveContainer" containerID="f22b43db4e9def85b7feb2078fbd68afb9cb74aeb772c92dfe946c6a1ca1ac22" Oct 30 18:15:05.211024 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:05.211005 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds_odh-ai-gateway-infra(3c689fe2-b26b-49f0-b37b-902238098837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" podUID="3c689fe2-b26b-49f0-b37b-902238098837" Oct 30 18:15:05.212349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.212327 4883 scope.go:122] "RemoveContainer" containerID="95368ca64cdf06367c6906a272dae06548d4f904f4ef5b2518c0c4c2d246b8ba" Oct 30 18:15:05.212701 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:05.212677 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw_odh-ai-gateway-infra(cd6fb8b2-f083-490e-8ed7-c96a99f01b22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" Oct 30 18:15:05.214088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.214067 4883 scope.go:122] "RemoveContainer" containerID="e079f946f6031f9a252b2b9c5598d5b2ef38b1b9c79f884a5d1c0b06e633d4f3" Oct 30 18:15:05.214405 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:05.214381 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57_odh-ai-gateway-infra(1021fc1b-74a1-49ae-865c-5d357bfb2c25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" Oct 30 18:15:05.215819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.215798 4883 generic.go:356] "Generic (PLEG): container finished" podID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" containerID="0974b2d4d1daef0519993e990c244c69a5c6a6a21cd83eb85b91bf45e05d34fb" exitCode=60 Oct 30 18:15:05.215906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.215866 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" event={"ID":"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3","Type":"ContainerDied","Data":"0974b2d4d1daef0519993e990c244c69a5c6a6a21cd83eb85b91bf45e05d34fb"} Oct 30 18:15:05.216168 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.216152 4883 scope.go:122] "RemoveContainer" containerID="0974b2d4d1daef0519993e990c244c69a5c6a6a21cd83eb85b91bf45e05d34fb" Oct 30 18:15:05.216457 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:05.216434 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp_odh-ai-gateway-infra(5e99ea40-c6f2-4bb2-a2c1-d62304538ec3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" Oct 30 18:15:05.222389 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.222309 4883 scope.go:122] "RemoveContainer" containerID="20b5896245bb8d1e4b27319556a9b733add2b58b4c66f7e159ce5554af623781" Oct 30 18:15:05.240980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:05.240771 4883 scope.go:122] "RemoveContainer" containerID="38b35aefa66aa018079ff6e9d200d023eb7898d183b0fec3cd968a417ec7b2e4" Oct 30 18:15:06.055064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:06.055029 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d"] Oct 30 18:15:06.068037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:06.067997 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7075388e-dq94d"] Oct 30 18:15:06.226014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:06.225987 4883 scope.go:122] "RemoveContainer" containerID="f22b43db4e9def85b7feb2078fbd68afb9cb74aeb772c92dfe946c6a1ca1ac22" Oct 30 18:15:06.226484 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:06.226389 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds_odh-ai-gateway-infra(3c689fe2-b26b-49f0-b37b-902238098837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" podUID="3c689fe2-b26b-49f0-b37b-902238098837" Oct 30 18:15:06.227872 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:06.227849 4883 scope.go:122] "RemoveContainer" containerID="0974b2d4d1daef0519993e990c244c69a5c6a6a21cd83eb85b91bf45e05d34fb" Oct 30 18:15:06.228234 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:06.228209 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp_odh-ai-gateway-infra(5e99ea40-c6f2-4bb2-a2c1-d62304538ec3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" Oct 30 18:15:06.229735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:06.229715 4883 scope.go:122] "RemoveContainer" containerID="fe5adcbf5f7629d9dfabd8b83852286f76da2dbfc775745464b9682caaf0eec4" Oct 30 18:15:06.230059 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:06.230037 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8_odh-ai-gateway-infra(7e786833-606f-4c49-9d09-56bef62b1a4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" Oct 30 18:15:07.179826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:07.179785 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb0e632-4a21-4909-9d4c-8eb6d0d8135c" path="/var/lib/kubelet/pods/2fb0e632-4a21-4909-9d4c-8eb6d0d8135c/volumes" Oct 30 18:15:08.074286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.074247 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8"] Oct 30 18:15:08.080280 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.080246 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.119475 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.119429 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8"] Oct 30 18:15:08.122193 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.122159 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c6b2120f-openshift-default-dockercfg-xt8vs\"" Oct 30 18:15:08.127307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.127263 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.127539 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.127322 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.127539 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.127451 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07abf63a-7209-4001-90dc-af2cb30525ec-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.127539 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.127519 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67b8v\" (UniqueName: \"kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-kube-api-access-67b8v\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.127722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.127560 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.127722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.127711 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07abf63a-7209-4001-90dc-af2cb30525ec-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.127823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.127766 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.127823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.127807 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.128007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.127982 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.229284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.229238 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.229284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.229293 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.229515 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.229322 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07abf63a-7209-4001-90dc-af2cb30525ec-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.229515 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.229353 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67b8v\" (UniqueName: \"kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-kube-api-access-67b8v\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.229515 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.229388 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.229659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.229529 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07abf63a-7209-4001-90dc-af2cb30525ec-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.229659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.229574 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.229659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.229611 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.229812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.229721 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.229812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.229733 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.229812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.229775 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.230099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.230067 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.230199 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.230108 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.230350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.230326 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07abf63a-7209-4001-90dc-af2cb30525ec-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.231833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.231814 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.232151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.232130 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07abf63a-7209-4001-90dc-af2cb30525ec-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.255926 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.255895 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.256126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.256102 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67b8v\" (UniqueName: \"kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-kube-api-access-67b8v\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.395597 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.395545 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:08.739099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.739063 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8"] Oct 30 18:15:08.739711 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:15:08.739681 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07abf63a_7209_4001_90dc_af2cb30525ec.slice/crio-317e73bc97ef67fd406e0e1dc1e6a6541a347051c24fbf5a55c4bdf78bad2b50 WatchSource:0}: Error finding container 317e73bc97ef67fd406e0e1dc1e6a6541a347051c24fbf5a55c4bdf78bad2b50: Status 404 returned error can't find the container with id 317e73bc97ef67fd406e0e1dc1e6a6541a347051c24fbf5a55c4bdf78bad2b50 Oct 30 18:15:08.742750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.742708 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:15:08.742862 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.742793 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:15:08.742862 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:08.742828 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:15:09.274958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:09.272986 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" event={"ID":"07abf63a-7209-4001-90dc-af2cb30525ec","Type":"ContainerStarted","Data":"2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a"} Oct 30 18:15:09.274958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:09.273060 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" event={"ID":"07abf63a-7209-4001-90dc-af2cb30525ec","Type":"ContainerStarted","Data":"317e73bc97ef67fd406e0e1dc1e6a6541a347051c24fbf5a55c4bdf78bad2b50"} Oct 30 18:15:09.377845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:09.377788 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" podStartSLOduration=1.377770404 podStartE2EDuration="1.377770404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:09.357495946 +0000 UTC m=+1368.314499783" watchObservedRunningTime="2026-10-30 18:15:09.377770404 +0000 UTC m=+1368.334774241" Oct 30 18:15:09.396306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:09.396258 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:09.398151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:09.398121 4883 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:15:09.398289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:09.398176 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" podUID="07abf63a-7209-4001-90dc-af2cb30525ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:15:10.396407 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:10.396364 4883 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:15:10.397076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:10.397038 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" podUID="07abf63a-7209-4001-90dc-af2cb30525ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:15:11.396174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:11.396133 4883 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:15:11.396408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:11.396189 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" podUID="07abf63a-7209-4001-90dc-af2cb30525ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:15:12.396718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:12.396679 4883 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:15:12.397197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:12.396738 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" podUID="07abf63a-7209-4001-90dc-af2cb30525ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:15:13.396554 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:13.396511 4883 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:15:13.396734 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:13.396568 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" podUID="07abf63a-7209-4001-90dc-af2cb30525ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:15:14.034094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:14.034057 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw"] Oct 30 18:15:14.046312 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:14.046278 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d6bcfc8-jzlmw"] Oct 30 18:15:14.396582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:14.396536 4883 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:15:14.396780 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:14.396613 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" podUID="07abf63a-7209-4001-90dc-af2cb30525ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:15:15.182128 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:15.182086 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19372b31-927c-4e58-b860-14b49d960fe7" path="/var/lib/kubelet/pods/19372b31-927c-4e58-b860-14b49d960fe7/volumes" Oct 30 18:15:15.183002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:15.182985 4883 scope.go:122] "RemoveContainer" containerID="f22b43db4e9def85b7feb2078fbd68afb9cb74aeb772c92dfe946c6a1ca1ac22" Oct 30 18:15:15.572049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:15.572015 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:15.573332 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:15.573309 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:15.575326 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:15.575294 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:15:16.031907 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:16.031859 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww"] Oct 30 18:15:16.036412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:16.036362 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7eb175f-df7ww"] Oct 30 18:15:16.172391 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:16.172346 4883 scope.go:122] "RemoveContainer" containerID="0974b2d4d1daef0519993e990c244c69a5c6a6a21cd83eb85b91bf45e05d34fb" Oct 30 18:15:16.174182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:16.174155 4883 scope.go:122] "RemoveContainer" containerID="e079f946f6031f9a252b2b9c5598d5b2ef38b1b9c79f884a5d1c0b06e633d4f3" Oct 30 18:15:16.174312 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:16.174265 4883 scope.go:122] "RemoveContainer" containerID="fe5adcbf5f7629d9dfabd8b83852286f76da2dbfc775745464b9682caaf0eec4" Oct 30 18:15:16.305434 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:16.305328 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq"] Oct 30 18:15:16.310046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:16.310006 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0498d2-twgmq"] Oct 30 18:15:16.354456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:16.354421 4883 generic.go:356] "Generic (PLEG): container finished" podID="3c689fe2-b26b-49f0-b37b-902238098837" containerID="bb7670455fbd0bbb1cbbae0a94f923a4bf81fd145ea90d0e54343283ce148067" exitCode=60 Oct 30 18:15:16.354643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:16.354493 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" event={"ID":"3c689fe2-b26b-49f0-b37b-902238098837","Type":"ContainerDied","Data":"bb7670455fbd0bbb1cbbae0a94f923a4bf81fd145ea90d0e54343283ce148067"} Oct 30 18:15:16.354643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:16.354595 4883 scope.go:122] "RemoveContainer" containerID="f22b43db4e9def85b7feb2078fbd68afb9cb74aeb772c92dfe946c6a1ca1ac22" Oct 30 18:15:16.355223 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:16.355204 4883 scope.go:122] "RemoveContainer" containerID="bb7670455fbd0bbb1cbbae0a94f923a4bf81fd145ea90d0e54343283ce148067" Oct 30 18:15:16.355526 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:16.355504 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds_odh-ai-gateway-infra(3c689fe2-b26b-49f0-b37b-902238098837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" podUID="3c689fe2-b26b-49f0-b37b-902238098837" Oct 30 18:15:17.043321 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.043269 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb"] Oct 30 18:15:17.055457 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.055420 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b8125f9-swpxb"] Oct 30 18:15:17.180614 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.180526 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0816c7e7-f68b-49f6-bd92-8632a42f9865" path="/var/lib/kubelet/pods/0816c7e7-f68b-49f6-bd92-8632a42f9865/volumes" Oct 30 18:15:17.181277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.181251 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb68642-421e-46c9-9d34-0e60f5694978" path="/var/lib/kubelet/pods/9eb68642-421e-46c9-9d34-0e60f5694978/volumes" Oct 30 18:15:17.181815 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.181799 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbbaf21a-9dc7-42da-abd2-7c6ed668963c" path="/var/lib/kubelet/pods/fbbaf21a-9dc7-42da-abd2-7c6ed668963c/volumes" Oct 30 18:15:17.364487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.364454 4883 generic.go:356] "Generic (PLEG): container finished" podID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" containerID="cd93aad617b9f44882bac0d7a05f392be2a5128a8ee84d72723af64a7ac84917" exitCode=60 Oct 30 18:15:17.364991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.364537 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" event={"ID":"1021fc1b-74a1-49ae-865c-5d357bfb2c25","Type":"ContainerDied","Data":"cd93aad617b9f44882bac0d7a05f392be2a5128a8ee84d72723af64a7ac84917"} Oct 30 18:15:17.364991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.364621 4883 scope.go:122] "RemoveContainer" containerID="e079f946f6031f9a252b2b9c5598d5b2ef38b1b9c79f884a5d1c0b06e633d4f3" Oct 30 18:15:17.365537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.365115 4883 scope.go:122] "RemoveContainer" containerID="cd93aad617b9f44882bac0d7a05f392be2a5128a8ee84d72723af64a7ac84917" Oct 30 18:15:17.365537 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:17.365497 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57_odh-ai-gateway-infra(1021fc1b-74a1-49ae-865c-5d357bfb2c25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" Oct 30 18:15:17.367571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.367532 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" event={"ID":"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3","Type":"ContainerStarted","Data":"d002ccaf400b04d7967d042594b9e2576a7c33f8e21aca685dd2d5cb307f0c4f"} Oct 30 18:15:17.370114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.370081 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" event={"ID":"7e786833-606f-4c49-9d09-56bef62b1a4a","Type":"ContainerStarted","Data":"25f13ce1f8cd72ee1f3629bf838f6a488d48001138dcb9455131b01097511563"} Oct 30 18:15:17.408984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.408870 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds"] Oct 30 18:15:17.608921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.608867 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" Oct 30 18:15:17.706304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.706203 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c689fe2-b26b-49f0-b37b-902238098837-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c689fe2-b26b-49f0-b37b-902238098837-maas-api-service-ca\") pod \"3c689fe2-b26b-49f0-b37b-902238098837\" (UID: \"3c689fe2-b26b-49f0-b37b-902238098837\") " Oct 30 18:15:17.706530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.706505 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c689fe2-b26b-49f0-b37b-902238098837-kube-api-access-szdxp\" (UniqueName: \"kubernetes.io/projected/3c689fe2-b26b-49f0-b37b-902238098837-kube-api-access-szdxp\") pod \"3c689fe2-b26b-49f0-b37b-902238098837\" (UID: \"3c689fe2-b26b-49f0-b37b-902238098837\") " Oct 30 18:15:17.706678 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.706657 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c689fe2-b26b-49f0-b37b-902238098837-maas-api-service-ca" pod "3c689fe2-b26b-49f0-b37b-902238098837" (UID: "3c689fe2-b26b-49f0-b37b-902238098837"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:17.707622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.707592 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c689fe2-b26b-49f0-b37b-902238098837-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:15:17.708884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.708843 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c689fe2-b26b-49f0-b37b-902238098837-kube-api-access-szdxp" pod "3c689fe2-b26b-49f0-b37b-902238098837" (UID: "3c689fe2-b26b-49f0-b37b-902238098837"). InnerVolumeSpecName "kube-api-access-szdxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:17.808924 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:17.808862 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szdxp\" (UniqueName: \"kubernetes.io/projected/3c689fe2-b26b-49f0-b37b-902238098837-kube-api-access-szdxp\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:15:18.173348 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.172584 4883 scope.go:122] "RemoveContainer" containerID="95368ca64cdf06367c6906a272dae06548d4f904f4ef5b2518c0c4c2d246b8ba" Oct 30 18:15:18.382211 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.382173 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" Oct 30 18:15:18.382211 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.382186 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds" event={"ID":"3c689fe2-b26b-49f0-b37b-902238098837","Type":"ContainerDied","Data":"623c9306bfdaf06ee29b7c7599e8376d22032c3d351bae65be1e2030128d5389"} Oct 30 18:15:18.382750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.382254 4883 scope.go:122] "RemoveContainer" containerID="bb7670455fbd0bbb1cbbae0a94f923a4bf81fd145ea90d0e54343283ce148067" Oct 30 18:15:18.386615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.386589 4883 generic.go:356] "Generic (PLEG): container finished" podID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" containerID="d002ccaf400b04d7967d042594b9e2576a7c33f8e21aca685dd2d5cb307f0c4f" exitCode=60 Oct 30 18:15:18.386750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.386661 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" event={"ID":"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3","Type":"ContainerDied","Data":"d002ccaf400b04d7967d042594b9e2576a7c33f8e21aca685dd2d5cb307f0c4f"} Oct 30 18:15:18.387874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.387192 4883 scope.go:122] "RemoveContainer" containerID="d002ccaf400b04d7967d042594b9e2576a7c33f8e21aca685dd2d5cb307f0c4f" Oct 30 18:15:18.387874 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:18.387575 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp_odh-ai-gateway-infra(5e99ea40-c6f2-4bb2-a2c1-d62304538ec3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" Oct 30 18:15:18.389056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.389036 4883 generic.go:356] "Generic (PLEG): container finished" podID="7e786833-606f-4c49-9d09-56bef62b1a4a" containerID="25f13ce1f8cd72ee1f3629bf838f6a488d48001138dcb9455131b01097511563" exitCode=60 Oct 30 18:15:18.389146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.389064 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" event={"ID":"7e786833-606f-4c49-9d09-56bef62b1a4a","Type":"ContainerDied","Data":"25f13ce1f8cd72ee1f3629bf838f6a488d48001138dcb9455131b01097511563"} Oct 30 18:15:18.389456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.389437 4883 scope.go:122] "RemoveContainer" containerID="25f13ce1f8cd72ee1f3629bf838f6a488d48001138dcb9455131b01097511563" Oct 30 18:15:18.389810 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:18.389778 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8_odh-ai-gateway-infra(7e786833-606f-4c49-9d09-56bef62b1a4a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" Oct 30 18:15:18.394785 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.394760 4883 scope.go:122] "RemoveContainer" containerID="0974b2d4d1daef0519993e990c244c69a5c6a6a21cd83eb85b91bf45e05d34fb" Oct 30 18:15:18.410118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.410096 4883 scope.go:122] "RemoveContainer" containerID="fe5adcbf5f7629d9dfabd8b83852286f76da2dbfc775745464b9682caaf0eec4" Oct 30 18:15:18.422501 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.422468 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57"] Oct 30 18:15:18.447386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.447344 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds"] Oct 30 18:15:18.451413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.451376 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889735-9s7ds"] Oct 30 18:15:18.463477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.463440 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp"] Oct 30 18:15:18.564762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.564717 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8"] Oct 30 18:15:18.632924 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.632891 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" Oct 30 18:15:18.723019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.722879 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1021fc1b-74a1-49ae-865c-5d357bfb2c25-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1021fc1b-74a1-49ae-865c-5d357bfb2c25-maas-api-service-ca\") pod \"1021fc1b-74a1-49ae-865c-5d357bfb2c25\" (UID: \"1021fc1b-74a1-49ae-865c-5d357bfb2c25\") " Oct 30 18:15:18.723230 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.723209 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1021fc1b-74a1-49ae-865c-5d357bfb2c25-kube-api-access-gsnr7\" (UniqueName: \"kubernetes.io/projected/1021fc1b-74a1-49ae-865c-5d357bfb2c25-kube-api-access-gsnr7\") pod \"1021fc1b-74a1-49ae-865c-5d357bfb2c25\" (UID: \"1021fc1b-74a1-49ae-865c-5d357bfb2c25\") " Oct 30 18:15:18.723337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.723292 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1021fc1b-74a1-49ae-865c-5d357bfb2c25-maas-api-service-ca" pod "1021fc1b-74a1-49ae-865c-5d357bfb2c25" (UID: "1021fc1b-74a1-49ae-865c-5d357bfb2c25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:18.723705 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.723682 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1021fc1b-74a1-49ae-865c-5d357bfb2c25-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:15:18.725733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.725706 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1021fc1b-74a1-49ae-865c-5d357bfb2c25-kube-api-access-gsnr7" pod "1021fc1b-74a1-49ae-865c-5d357bfb2c25" (UID: "1021fc1b-74a1-49ae-865c-5d357bfb2c25"). InnerVolumeSpecName "kube-api-access-gsnr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:18.825405 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:18.825368 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsnr7\" (UniqueName: \"kubernetes.io/projected/1021fc1b-74a1-49ae-865c-5d357bfb2c25-kube-api-access-gsnr7\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:15:19.181327 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.181293 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c689fe2-b26b-49f0-b37b-902238098837" path="/var/lib/kubelet/pods/3c689fe2-b26b-49f0-b37b-902238098837/volumes" Oct 30 18:15:19.398826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.398792 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" Oct 30 18:15:19.399398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.398827 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57" event={"ID":"1021fc1b-74a1-49ae-865c-5d357bfb2c25","Type":"ContainerDied","Data":"cce94e7c4283b22c69fbca1d80cc4d222dd8b5078607ac5075119a4f158bdb61"} Oct 30 18:15:19.399398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.398906 4883 scope.go:122] "RemoveContainer" containerID="cd93aad617b9f44882bac0d7a05f392be2a5128a8ee84d72723af64a7ac84917" Oct 30 18:15:19.413631 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.413589 4883 generic.go:356] "Generic (PLEG): container finished" podID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" containerID="ee63998e06ca3a5d541ae521cd8272ce1098dc40ae9afe614625d901ea06e393" exitCode=60 Oct 30 18:15:19.413819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.413640 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" event={"ID":"cd6fb8b2-f083-490e-8ed7-c96a99f01b22","Type":"ContainerDied","Data":"ee63998e06ca3a5d541ae521cd8272ce1098dc40ae9afe614625d901ea06e393"} Oct 30 18:15:19.414205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.414181 4883 scope.go:122] "RemoveContainer" containerID="ee63998e06ca3a5d541ae521cd8272ce1098dc40ae9afe614625d901ea06e393" Oct 30 18:15:19.414649 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:19.414624 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw_odh-ai-gateway-infra(cd6fb8b2-f083-490e-8ed7-c96a99f01b22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" Oct 30 18:15:19.415015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.414994 4883 scope.go:122] "RemoveContainer" containerID="95368ca64cdf06367c6906a272dae06548d4f904f4ef5b2518c0c4c2d246b8ba" Oct 30 18:15:19.458011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.457970 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57"] Oct 30 18:15:19.465430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.465389 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889735-7xd57"] Oct 30 18:15:19.644058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.644015 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" Oct 30 18:15:19.682523 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.682446 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" Oct 30 18:15:19.740016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.739975 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-maas-api-service-ca\") pod \"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3\" (UID: \"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3\") " Oct 30 18:15:19.740215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.740180 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-kube-api-access-2bfjz\" (UniqueName: \"kubernetes.io/projected/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-kube-api-access-2bfjz\") pod \"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3\" (UID: \"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3\") " Oct 30 18:15:19.740408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.740380 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-maas-api-service-ca" pod "5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" (UID: "5e99ea40-c6f2-4bb2-a2c1-d62304538ec3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:19.740792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.740768 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:15:19.745985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.742745 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-kube-api-access-2bfjz" pod "5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" (UID: "5e99ea40-c6f2-4bb2-a2c1-d62304538ec3"). InnerVolumeSpecName "kube-api-access-2bfjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:19.841424 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.841387 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e786833-606f-4c49-9d09-56bef62b1a4a-kube-api-access-229ts\" (UniqueName: \"kubernetes.io/projected/7e786833-606f-4c49-9d09-56bef62b1a4a-kube-api-access-229ts\") pod \"7e786833-606f-4c49-9d09-56bef62b1a4a\" (UID: \"7e786833-606f-4c49-9d09-56bef62b1a4a\") " Oct 30 18:15:19.841650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.841469 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e786833-606f-4c49-9d09-56bef62b1a4a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e786833-606f-4c49-9d09-56bef62b1a4a-maas-api-service-ca\") pod \"7e786833-606f-4c49-9d09-56bef62b1a4a\" (UID: \"7e786833-606f-4c49-9d09-56bef62b1a4a\") " Oct 30 18:15:19.841981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.841905 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e786833-606f-4c49-9d09-56bef62b1a4a-maas-api-service-ca" pod "7e786833-606f-4c49-9d09-56bef62b1a4a" (UID: "7e786833-606f-4c49-9d09-56bef62b1a4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:19.842158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.842140 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e786833-606f-4c49-9d09-56bef62b1a4a-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:15:19.842214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.842163 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bfjz\" (UniqueName: \"kubernetes.io/projected/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3-kube-api-access-2bfjz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:15:19.843743 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.843715 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e786833-606f-4c49-9d09-56bef62b1a4a-kube-api-access-229ts" pod "7e786833-606f-4c49-9d09-56bef62b1a4a" (UID: "7e786833-606f-4c49-9d09-56bef62b1a4a"). InnerVolumeSpecName "kube-api-access-229ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:19.943733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:19.943630 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-229ts\" (UniqueName: \"kubernetes.io/projected/7e786833-606f-4c49-9d09-56bef62b1a4a-kube-api-access-229ts\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:15:20.426196 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.426160 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" Oct 30 18:15:20.426196 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.426182 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp" event={"ID":"5e99ea40-c6f2-4bb2-a2c1-d62304538ec3","Type":"ContainerDied","Data":"7373ff85da40a1e79de689dda8ef016188b59108c2f8984b3e3edd6e2fed0cad"} Oct 30 18:15:20.427084 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.426249 4883 scope.go:122] "RemoveContainer" containerID="d002ccaf400b04d7967d042594b9e2576a7c33f8e21aca685dd2d5cb307f0c4f" Oct 30 18:15:20.429353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.429329 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" Oct 30 18:15:20.429440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.429349 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8" event={"ID":"7e786833-606f-4c49-9d09-56bef62b1a4a","Type":"ContainerDied","Data":"9ed5f47998200d832e4efc3d1a9506f563eab458be8e576aef2e3d7e6544d5d0"} Oct 30 18:15:20.442363 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.442334 4883 scope.go:122] "RemoveContainer" containerID="25f13ce1f8cd72ee1f3629bf838f6a488d48001138dcb9455131b01097511563" Oct 30 18:15:20.484794 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.484758 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp"] Oct 30 18:15:20.489317 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.489279 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c6b2120f-29889735-8mbqp"] Oct 30 18:15:20.528058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.528023 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8"] Oct 30 18:15:20.538292 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.538256 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889735-fgcm8"] Oct 30 18:15:20.563499 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.563460 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw"] Oct 30 18:15:20.794976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.794948 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" Oct 30 18:15:20.974765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.974724 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-kube-api-access-nzncj\" (UniqueName: \"kubernetes.io/projected/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-kube-api-access-nzncj\") pod \"cd6fb8b2-f083-490e-8ed7-c96a99f01b22\" (UID: \"cd6fb8b2-f083-490e-8ed7-c96a99f01b22\") " Oct 30 18:15:20.975075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.975056 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-maas-api-service-ca\") pod \"cd6fb8b2-f083-490e-8ed7-c96a99f01b22\" (UID: \"cd6fb8b2-f083-490e-8ed7-c96a99f01b22\") " Oct 30 18:15:20.975437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.975403 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-maas-api-service-ca" pod "cd6fb8b2-f083-490e-8ed7-c96a99f01b22" (UID: "cd6fb8b2-f083-490e-8ed7-c96a99f01b22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:20.975632 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.975617 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:15:20.977078 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:20.977046 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-kube-api-access-nzncj" pod "cd6fb8b2-f083-490e-8ed7-c96a99f01b22" (UID: "cd6fb8b2-f083-490e-8ed7-c96a99f01b22"). InnerVolumeSpecName "kube-api-access-nzncj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:21.077286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:21.077251 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzncj\" (UniqueName: \"kubernetes.io/projected/cd6fb8b2-f083-490e-8ed7-c96a99f01b22-kube-api-access-nzncj\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.188304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:21.188261 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" path="/var/lib/kubelet/pods/1021fc1b-74a1-49ae-865c-5d357bfb2c25/volumes" Oct 30 18:15:21.188768 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:21.188747 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" path="/var/lib/kubelet/pods/5e99ea40-c6f2-4bb2-a2c1-d62304538ec3/volumes" Oct 30 18:15:21.189318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:21.189296 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" path="/var/lib/kubelet/pods/7e786833-606f-4c49-9d09-56bef62b1a4a/volumes" Oct 30 18:15:21.452012 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:21.451966 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" event={"ID":"cd6fb8b2-f083-490e-8ed7-c96a99f01b22","Type":"ContainerDied","Data":"ec4c5eb8118565ece8781fc8af2466c9b7f8e1cdbb7a239f419b7cb9ed7681d0"} Oct 30 18:15:21.452526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:21.452017 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw" Oct 30 18:15:21.452526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:21.452093 4883 scope.go:122] "RemoveContainer" containerID="ee63998e06ca3a5d541ae521cd8272ce1098dc40ae9afe614625d901ea06e393" Oct 30 18:15:21.510984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:21.510947 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw"] Oct 30 18:15:21.523304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:21.523266 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-57ee15-29889735-p5fsw"] Oct 30 18:15:21.653462 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:21.653420 4883 scope.go:122] "RemoveContainer" containerID="e8135e5d76b379e82508cb2d4055c57a19d974d1cffb2ffb1587e2583f933f51" Oct 30 18:15:21.664581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:21.664538 4883 scope.go:122] "RemoveContainer" containerID="ba0750e925694fa9af6d14b5983309f33fefc24e0558fd0013754caa38efbe0e" Oct 30 18:15:23.179717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:23.179683 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" path="/var/lib/kubelet/pods/cd6fb8b2-f083-490e-8ed7-c96a99f01b22/volumes" Oct 30 18:15:27.033177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:27.033137 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd"] Oct 30 18:15:27.046599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:27.046557 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t"] Oct 30 18:15:27.050564 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:27.050519 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ca3714-2kjjd"] Oct 30 18:15:27.054269 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:27.054243 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555cb312-wxk9t"] Oct 30 18:15:27.178843 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:27.178795 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4" path="/var/lib/kubelet/pods/2d2b59af-517b-4b0b-aaf4-0d8c77a3f7a4/volumes" Oct 30 18:15:27.179428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:27.179403 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f23f75f-fd4c-4592-84d5-ef0a20be658e" path="/var/lib/kubelet/pods/5f23f75f-fd4c-4592-84d5-ef0a20be658e/volumes" Oct 30 18:15:30.031440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:30.031397 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv"] Oct 30 18:15:30.044869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:30.044831 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb5547d1-d7mjv"] Oct 30 18:15:31.181444 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:31.181406 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e2118c-04b2-4055-b486-a5a15f981af9" path="/var/lib/kubelet/pods/77e2118c-04b2-4055-b486-a5a15f981af9/volumes" Oct 30 18:15:35.135114 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:15:35.134894 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:15:43.031372 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:43.031336 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92"] Oct 30 18:15:43.034968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:43.034919 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ea3406-tds92"] Oct 30 18:15:43.179564 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:43.179520 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c" path="/var/lib/kubelet/pods/9c8cb9c9-a632-4ffa-8a9b-9c2eb1ea6c6c/volumes" Oct 30 18:15:47.045899 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:47.045855 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl"] Oct 30 18:15:47.057884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:47.057836 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0af5393-clgnl"] Oct 30 18:15:47.203204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:47.203137 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ce6cf1-12ff-4a5b-b7b9-36527ef10795" path="/var/lib/kubelet/pods/b4ce6cf1-12ff-4a5b-b7b9-36527ef10795/volumes" Oct 30 18:15:53.033672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:53.033632 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w"] Oct 30 18:15:53.048387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:53.048345 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8fc3315-zvq4w"] Oct 30 18:15:53.180433 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:53.180389 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e4a4d5-a678-4deb-9639-221db9eca09a" path="/var/lib/kubelet/pods/a3e4a4d5-a678-4deb-9639-221db9eca09a/volumes" Oct 30 18:15:57.231823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:57.231765 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s"] Oct 30 18:15:57.232748 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:15:57.232675 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" podUID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" containerName="istio-proxy" containerID="cri-o://290c6642f292a3a3d5f989ca87da6d6b240d2aaf0413d6f8dfa7b56d78d6c9e9" gracePeriod=30 Oct 30 18:16:00.263977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.260056 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.371311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.352038 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.371311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.353195 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:16:00.477183 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.477123 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:16:00.531811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.530504 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.531811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.530919 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.531811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.531222 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Oct 30 18:16:00.552056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.542700 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 18:16:00.552056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.542815 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:16:00.587982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.582947 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.701184 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.701116 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.867093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.867040 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:16:00.907236 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.907183 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.981282 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:00.981230 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.353647 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.352072 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.353647 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.352366 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.482666 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.482587 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.539963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.534109 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:16:01.558978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.558727 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.576374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.576053 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:16:01.602999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.589240 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.603810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.603709 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.689101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.688066 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:16:01.689101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.688135 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:16:01.689101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.688167 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:16:01.689101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.688199 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:16:01.689101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.688231 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:16:01.689101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.688262 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:16:01.689101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.688293 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:16:01.736157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.736107 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.736555 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.736529 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:16:01.881022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.874969 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:01.881918 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.881888 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:16:01.940960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.923044 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:16:01.940960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:01.924578 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:16:02.415042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.379358 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:02.415042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.379802 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:02.415042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.380020 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:16:02.551645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.531052 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:02.551645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.531109 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:02.551645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.531046 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:02.573976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.571037 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:16:02.959178 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.536021 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:02.962549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.601782 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:02.969277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.965247 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:16:02.969277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.611178 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:02.969277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.612498 4883 patch_prober.go:29] interesting pod/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:16:02.969277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.965816 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" podUID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:16:02.969277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.843371 4883 generic.go:356] "Generic (PLEG): container finished" podID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" containerID="290c6642f292a3a3d5f989ca87da6d6b240d2aaf0413d6f8dfa7b56d78d6c9e9" exitCode=0 Oct 30 18:16:02.969277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.843402 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" event={"ID":"269010f9-cf3f-48ee-ba1d-8fdb5404a937","Type":"ContainerDied","Data":"290c6642f292a3a3d5f989ca87da6d6b240d2aaf0413d6f8dfa7b56d78d6c9e9"} Oct 30 18:16:02.969277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.966095 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:16:02.969277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.754282 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:02.969277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.882555 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:16:02.989962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.601273 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:16:02.989962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:02.987526 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.390345 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:03.390296 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.391583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:03.391161 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:16:03.391823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:03.391798 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:16:03.532578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:03.532520 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:16:03.610627 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:03.610579 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.683314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:03.666068 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:03.683314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:03.666136 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:03.683314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:03.667902 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:16:03.932579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:03.931922 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk"] Oct 30 18:16:03.978308 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:03.956763 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfb2fdd-gxklk"] Oct 30 18:16:04.094529 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.007095 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.116006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.007135 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:16:04.121284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.007158 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:16:04.121923 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.007287 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:16:04.140561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.008585 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:16:04.140561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.008613 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:16:04.140561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.017519 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:16:04.140561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.017558 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:16:04.140561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.138927 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:16:04.388410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.388053 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.401967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.401531 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.402171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.401973 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:16:04.461968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.456064 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.540112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.540057 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.939050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.913807 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:04.983410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.983352 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:04.983704 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:04.983672 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.089119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.089019 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:05.089398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.089366 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.131962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.105725 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.171959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.126150 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.171959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.126295 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:16:05.203657 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.193592 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.269997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.194011 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.313007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.194179 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:16:05.354703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.194410 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:16:05.356195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.194616 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:16:05.521518 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.499340 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:16:05.552972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.550514 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:16:05.704085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.639232 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:16:05.704085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.657206 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.704085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.671091 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8632419b-f939-49f2-854a-a1b3903d8560" path="/var/lib/kubelet/pods/8632419b-f939-49f2-854a-a1b3903d8560/volumes" Oct 30 18:16:05.728967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.721312 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" event={"ID":"269010f9-cf3f-48ee-ba1d-8fdb5404a937","Type":"ContainerDied","Data":"e561b5ba6b0da55a1e0b0eed17157731d42b0c9cd2c89022110e2899e809bdf7"} Oct 30 18:16:05.728967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.721393 4883 scope.go:122] "RemoveContainer" containerID="290c6642f292a3a3d5f989ca87da6d6b240d2aaf0413d6f8dfa7b56d78d6c9e9" Oct 30 18:16:05.728967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.721605 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s" Oct 30 18:16:05.828961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.828598 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istiod-ca-cert\") pod \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " Oct 30 18:16:05.828961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.828821 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-kube-api-access-88c92\" (UniqueName: \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-kube-api-access-88c92\") pod \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " Oct 30 18:16:05.828961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.828864 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-token\" (UniqueName: \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-token\") pod \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " Oct 30 18:16:05.828961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.828894 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-data\") pod \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " Oct 30 18:16:05.828961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.828958 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-podinfo\") pod \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " Oct 30 18:16:05.829395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.829019 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-certs\") pod \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " Oct 30 18:16:05.829395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.829109 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-envoy\") pod \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " Oct 30 18:16:05.829395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.829138 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-socket\") pod \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " Oct 30 18:16:05.829395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.829240 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-credential-socket\") pod \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " Oct 30 18:16:05.899961 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:16:05.892154 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:16:05.924067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.914879 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istiod-ca-cert" pod "269010f9-cf3f-48ee-ba1d-8fdb5404a937" (UID: "269010f9-cf3f-48ee-ba1d-8fdb5404a937"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:05.924299 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.915114 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:16:05.929069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.929022 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:05.981021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.960079 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-certs" pod "269010f9-cf3f-48ee-ba1d-8fdb5404a937" (UID: "269010f9-cf3f-48ee-ba1d-8fdb5404a937"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:05.981396 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:05.981355 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-credential-socket" pod "269010f9-cf3f-48ee-ba1d-8fdb5404a937" (UID: "269010f9-cf3f-48ee-ba1d-8fdb5404a937"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.124190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.064478 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-socket" pod "269010f9-cf3f-48ee-ba1d-8fdb5404a937" (UID: "269010f9-cf3f-48ee-ba1d-8fdb5404a937"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.124190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.083021 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded" Oct 30 18:16:06.124190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.083340 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-token" pod "269010f9-cf3f-48ee-ba1d-8fdb5404a937" (UID: "269010f9-cf3f-48ee-ba1d-8fdb5404a937"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:06.124190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.083868 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-data" pod "269010f9-cf3f-48ee-ba1d-8fdb5404a937" (UID: "269010f9-cf3f-48ee-ba1d-8fdb5404a937"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.124190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.085196 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.124190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.085235 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.124190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.085248 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.124190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.085261 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.124190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.085273 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.124190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.085297 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.162302 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.161206 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:16:06.172753 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.172684 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:06.186178 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.186115 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:16:06.190976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.188857 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-kube-api-access-88c92" pod "269010f9-cf3f-48ee-ba1d-8fdb5404a937" (UID: "269010f9-cf3f-48ee-ba1d-8fdb5404a937"). InnerVolumeSpecName "kube-api-access-88c92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:06.204245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.203134 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-kube-api-access-88c92\" (UniqueName: \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-kube-api-access-88c92\") pod \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\" (UID: \"269010f9-cf3f-48ee-ba1d-8fdb5404a937\") " Oct 30 18:16:06.204245 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:16:06.204110 4883 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/269010f9-cf3f-48ee-ba1d-8fdb5404a937/volumes/kubernetes.io~projected/kube-api-access-88c92 Oct 30 18:16:06.204245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.204168 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-kube-api-access-88c92" pod "269010f9-cf3f-48ee-ba1d-8fdb5404a937" (UID: "269010f9-cf3f-48ee-ba1d-8fdb5404a937"). InnerVolumeSpecName "kube-api-access-88c92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:06.239246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.236231 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:16:06.239246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.237775 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-podinfo" pod "269010f9-cf3f-48ee-ba1d-8fdb5404a937" (UID: "269010f9-cf3f-48ee-ba1d-8fdb5404a937"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:16:06.302595 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.291718 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8"] Oct 30 18:16:06.302595 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.292646 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-envoy" pod "269010f9-cf3f-48ee-ba1d-8fdb5404a937" (UID: "269010f9-cf3f-48ee-ba1d-8fdb5404a937"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:16:06.319958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.305377 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.319958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.305425 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269010f9-cf3f-48ee-ba1d-8fdb5404a937-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.319958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.305440 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88c92\" (UniqueName: \"kubernetes.io/projected/269010f9-cf3f-48ee-ba1d-8fdb5404a937-kube-api-access-88c92\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:06.319958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.305488 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:16:06.319958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.306814 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:16:06.327689 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.327648 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a984a3c4-7g9c8"] Oct 30 18:16:06.410963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.376122 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:16:06.411604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.407355 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:16:06.525382 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.525323 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:06.624074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.624023 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s"] Oct 30 18:16:06.698990 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.698461 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:06.699201 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.699067 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:16:06.699384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.699355 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:06.743958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.716651 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0bdf03-openshift-default-6f6d647fb5-k9t5s"] Oct 30 18:16:06.919224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.919092 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:06.929229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.929172 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:06.930724 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:06.930688 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:07.048443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.045596 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:16:07.048443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.047141 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:16:07.059346 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.053496 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:16:07.198714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.195493 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:16:07.250067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.250020 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:07.324668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.324625 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:16:07.348994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.348452 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:07.368503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.368465 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1617a80d-af7d-4904-89cb-8c53cd8bd5b8" path="/var/lib/kubelet/pods/1617a80d-af7d-4904-89cb-8c53cd8bd5b8/volumes" Oct 30 18:16:07.369619 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.369069 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" path="/var/lib/kubelet/pods/269010f9-cf3f-48ee-ba1d-8fdb5404a937/volumes" Oct 30 18:16:07.371007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.370405 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:16:07.371007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.370460 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:16:07.371007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.370493 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:16:07.371007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.370526 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:16:07.371007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.370559 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:16:07.373808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.373400 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:16:07.398316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.398087 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:16:07.401745 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.401716 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:16:07.408427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.408379 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:16:07.416608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:07.416575 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:16:21.740746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:21.740711 4883 scope.go:122] "RemoveContainer" containerID="bc0e1c32aa657bfdd254f2568798730f014b81445c9b95717336b53198eb2312" Oct 30 18:16:21.751308 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:21.751277 4883 scope.go:122] "RemoveContainer" containerID="9fbaf758eda3c4b614e5d74386f246ae8f379c401a880b4154866897fb17a5f9" Oct 30 18:16:21.762583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:21.762559 4883 scope.go:122] "RemoveContainer" containerID="06a89084991f43e24bdafd29d19d2944a1669d01874f643b94f48d6fe019c6df" Oct 30 18:16:21.772719 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:21.772688 4883 scope.go:122] "RemoveContainer" containerID="a6672df9246fc50af99255782069cbe0c0e478f68ba96460e4fefcc4f344dd63" Oct 30 18:16:21.783199 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:21.783177 4883 scope.go:122] "RemoveContainer" containerID="0af28b10e433fda33826038fd58d8bc9ef60a2aa79b4c6bf88c073f7a3778540" Oct 30 18:16:21.793609 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:21.793583 4883 scope.go:122] "RemoveContainer" containerID="bffed7ea29f11ac814ec1d39cb1b9b569eaf4d3015f527b04a5ef6832eb74a00" Oct 30 18:16:27.030592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:27.030554 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh"] Oct 30 18:16:27.043390 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:27.043351 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30372d63-l5hwh"] Oct 30 18:16:27.179294 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:27.179259 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f010b93c-e961-4499-a3bc-efbf9588d883" path="/var/lib/kubelet/pods/f010b93c-e961-4499-a3bc-efbf9588d883/volumes" Oct 30 18:16:35.030518 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:35.030467 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8"] Oct 30 18:16:35.043812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:35.043768 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48677ed6-vgtp8"] Oct 30 18:16:35.080344 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:16:35.080304 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:16:35.179213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:35.179178 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8" path="/var/lib/kubelet/pods/86ee8276-dd4b-4b4e-bd9c-8f56bb8f27e8/volumes" Oct 30 18:16:44.036121 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:44.036076 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh"] Oct 30 18:16:44.044905 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:44.044861 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546c5d31-kv6rh"] Oct 30 18:16:45.028981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:45.028925 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th"] Oct 30 18:16:45.041946 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:45.041896 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f12911a-kh9th"] Oct 30 18:16:45.180176 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:45.180138 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec110c06-af6a-40a4-9423-9f8b661fa70d" path="/var/lib/kubelet/pods/ec110c06-af6a-40a4-9423-9f8b661fa70d/volumes" Oct 30 18:16:45.180664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:45.180648 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa42df06-1e0b-4380-a785-0640e61cf872" path="/var/lib/kubelet/pods/fa42df06-1e0b-4380-a785-0640e61cf872/volumes" Oct 30 18:16:49.772638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.772598 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs"] Oct 30 18:16:49.773205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773190 4883 cpu_manager.go:414] "containerMap: removing container" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" containerName="cleanup" Oct 30 18:16:49.773262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773206 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" containerName="cleanup" Oct 30 18:16:49.773262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773220 4883 cpu_manager.go:414] "containerMap: removing container" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" containerName="cleanup" Oct 30 18:16:49.773262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773225 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" containerName="cleanup" Oct 30 18:16:49.773262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773237 4883 cpu_manager.go:414] "containerMap: removing container" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" containerName="cleanup" Oct 30 18:16:49.773262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773242 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" containerName="cleanup" Oct 30 18:16:49.773262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773248 4883 cpu_manager.go:414] "containerMap: removing container" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" containerName="cleanup" Oct 30 18:16:49.773262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773252 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" containerName="cleanup" Oct 30 18:16:49.773262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773262 4883 cpu_manager.go:414] "containerMap: removing container" podUID="3c689fe2-b26b-49f0-b37b-902238098837" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773268 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c689fe2-b26b-49f0-b37b-902238098837" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773281 4883 cpu_manager.go:414] "containerMap: removing container" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773286 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773293 4883 cpu_manager.go:414] "containerMap: removing container" podUID="3c689fe2-b26b-49f0-b37b-902238098837" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773297 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c689fe2-b26b-49f0-b37b-902238098837" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773306 4883 cpu_manager.go:414] "containerMap: removing container" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773312 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773323 4883 cpu_manager.go:414] "containerMap: removing container" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773327 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773340 4883 cpu_manager.go:414] "containerMap: removing container" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773344 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773352 4883 cpu_manager.go:414] "containerMap: removing container" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773356 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773369 4883 cpu_manager.go:414] "containerMap: removing container" podUID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" containerName="istio-proxy" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773376 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" containerName="istio-proxy" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773384 4883 cpu_manager.go:414] "containerMap: removing container" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773388 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773467 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773476 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c689fe2-b26b-49f0-b37b-902238098837" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773485 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773493 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773498 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e99ea40-c6f2-4bb2-a2c1-d62304538ec3" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773505 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773511 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773517 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="269010f9-cf3f-48ee-ba1d-8fdb5404a937" containerName="istio-proxy" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773523 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e786833-606f-4c49-9d09-56bef62b1a4a" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773529 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773534 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773539 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773545 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c689fe2-b26b-49f0-b37b-902238098837" containerName="cleanup" Oct 30 18:16:49.773550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.773551 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" containerName="cleanup" Oct 30 18:16:49.777366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.777344 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:49.800058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.800021 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs"] Oct 30 18:16:49.829296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.829266 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:16:49.829509 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.829266 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:16:49.829509 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.829301 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:16:49.829650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.829312 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:16:49.837384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.837347 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs\" (UID: \"2ae34c41-6803-48b8-b0af-4e21bda88175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:49.837594 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.837456 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs\" (UID: \"2ae34c41-6803-48b8-b0af-4e21bda88175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:49.837594 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.837486 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-629gg\" (UniqueName: \"kubernetes.io/projected/2ae34c41-6803-48b8-b0af-4e21bda88175-kube-api-access-629gg\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs\" (UID: \"2ae34c41-6803-48b8-b0af-4e21bda88175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:49.938729 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.938688 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs\" (UID: \"2ae34c41-6803-48b8-b0af-4e21bda88175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:49.939019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.938807 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs\" (UID: \"2ae34c41-6803-48b8-b0af-4e21bda88175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:49.939019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.938838 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-629gg\" (UniqueName: \"kubernetes.io/projected/2ae34c41-6803-48b8-b0af-4e21bda88175-kube-api-access-629gg\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs\" (UID: \"2ae34c41-6803-48b8-b0af-4e21bda88175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:49.939502 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.939471 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs\" (UID: \"2ae34c41-6803-48b8-b0af-4e21bda88175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:49.941376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.941342 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs\" (UID: \"2ae34c41-6803-48b8-b0af-4e21bda88175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:49.963676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:49.963638 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-629gg\" (UniqueName: \"kubernetes.io/projected/2ae34c41-6803-48b8-b0af-4e21bda88175-kube-api-access-629gg\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs\" (UID: \"2ae34c41-6803-48b8-b0af-4e21bda88175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:50.092752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:50.092719 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:50.400869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:50.400845 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs"] Oct 30 18:16:50.402773 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:16:50.402740 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae34c41_6803_48b8_b0af_4e21bda88175.slice/crio-d2fcd388846e1c5d1a2edfd53f9b032e79d77c3f5ae7a2550828fefceca12d6e WatchSource:0}: Error finding container d2fcd388846e1c5d1a2edfd53f9b032e79d77c3f5ae7a2550828fefceca12d6e: Status 404 returned error can't find the container with id d2fcd388846e1c5d1a2edfd53f9b032e79d77c3f5ae7a2550828fefceca12d6e Oct 30 18:16:51.242644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:51.242609 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" event={"ID":"2ae34c41-6803-48b8-b0af-4e21bda88175","Type":"ContainerStarted","Data":"caa978a6dc036a1188809e7d6bf7113b4e537f7a7a5031587176b6aa96a41ebd"} Oct 30 18:16:51.243127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:51.242660 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" event={"ID":"2ae34c41-6803-48b8-b0af-4e21bda88175","Type":"ContainerStarted","Data":"d2fcd388846e1c5d1a2edfd53f9b032e79d77c3f5ae7a2550828fefceca12d6e"} Oct 30 18:16:51.287410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:51.287353 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" podStartSLOduration=2.287337625 podStartE2EDuration="2.287337625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:51.28520182 +0000 UTC m=+1470.242205656" watchObservedRunningTime="2026-10-30 18:16:51.287337625 +0000 UTC m=+1470.244341460" Oct 30 18:16:52.251409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:52.251370 4883 generic.go:356] "Generic (PLEG): container finished" podID="2ae34c41-6803-48b8-b0af-4e21bda88175" containerID="caa978a6dc036a1188809e7d6bf7113b4e537f7a7a5031587176b6aa96a41ebd" exitCode=0 Oct 30 18:16:52.251881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:52.251456 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" event={"ID":"2ae34c41-6803-48b8-b0af-4e21bda88175","Type":"ContainerDied","Data":"caa978a6dc036a1188809e7d6bf7113b4e537f7a7a5031587176b6aa96a41ebd"} Oct 30 18:16:53.434670 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:53.434636 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:53.489342 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:53.489307 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-xks-ca\") pod \"2ae34c41-6803-48b8-b0af-4e21bda88175\" (UID: \"2ae34c41-6803-48b8-b0af-4e21bda88175\") " Oct 30 18:16:53.489545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:53.489526 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae34c41-6803-48b8-b0af-4e21bda88175-kube-api-access-629gg\" (UniqueName: \"kubernetes.io/projected/2ae34c41-6803-48b8-b0af-4e21bda88175-kube-api-access-629gg\") pod \"2ae34c41-6803-48b8-b0af-4e21bda88175\" (UID: \"2ae34c41-6803-48b8-b0af-4e21bda88175\") " Oct 30 18:16:53.489658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:53.489642 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-openshift-ca\") pod \"2ae34c41-6803-48b8-b0af-4e21bda88175\" (UID: \"2ae34c41-6803-48b8-b0af-4e21bda88175\") " Oct 30 18:16:53.490069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:53.490044 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-openshift-ca" pod "2ae34c41-6803-48b8-b0af-4e21bda88175" (UID: "2ae34c41-6803-48b8-b0af-4e21bda88175"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:53.490635 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:53.490613 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:53.491786 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:53.491761 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae34c41-6803-48b8-b0af-4e21bda88175-kube-api-access-629gg" pod "2ae34c41-6803-48b8-b0af-4e21bda88175" (UID: "2ae34c41-6803-48b8-b0af-4e21bda88175"). InnerVolumeSpecName "kube-api-access-629gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:53.492080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:53.492054 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-xks-ca" pod "2ae34c41-6803-48b8-b0af-4e21bda88175" (UID: "2ae34c41-6803-48b8-b0af-4e21bda88175"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:53.591855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:53.591760 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-629gg\" (UniqueName: \"kubernetes.io/projected/2ae34c41-6803-48b8-b0af-4e21bda88175-kube-api-access-629gg\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:53.591855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:53.591797 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae34c41-6803-48b8-b0af-4e21bda88175-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:54.029810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.029766 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x"] Oct 30 18:16:54.033299 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.033269 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3173e1b-fbm5x"] Oct 30 18:16:54.239391 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.239355 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h"] Oct 30 18:16:54.239947 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.239917 4883 cpu_manager.go:414] "containerMap: removing container" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" containerName="cleanup" Oct 30 18:16:54.239947 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.239945 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1021fc1b-74a1-49ae-865c-5d357bfb2c25" containerName="cleanup" Oct 30 18:16:54.240044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.239957 4883 cpu_manager.go:414] "containerMap: removing container" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" containerName="cleanup" Oct 30 18:16:54.240044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.239962 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" containerName="cleanup" Oct 30 18:16:54.240044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.239980 4883 cpu_manager.go:414] "containerMap: removing container" podUID="3c689fe2-b26b-49f0-b37b-902238098837" containerName="cleanup" Oct 30 18:16:54.240044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.239986 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c689fe2-b26b-49f0-b37b-902238098837" containerName="cleanup" Oct 30 18:16:54.240044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.240001 4883 cpu_manager.go:414] "containerMap: removing container" podUID="2ae34c41-6803-48b8-b0af-4e21bda88175" containerName="revoke-keys" Oct 30 18:16:54.240044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.240006 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae34c41-6803-48b8-b0af-4e21bda88175" containerName="revoke-keys" Oct 30 18:16:54.240235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.240106 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae34c41-6803-48b8-b0af-4e21bda88175" containerName="revoke-keys" Oct 30 18:16:54.240235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.240117 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6fb8b2-f083-490e-8ed7-c96a99f01b22" containerName="cleanup" Oct 30 18:16:54.240235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.240131 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c689fe2-b26b-49f0-b37b-902238098837" containerName="cleanup" Oct 30 18:16:54.243767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.243742 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:54.263877 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.263843 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h"] Oct 30 18:16:54.270526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.270476 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" event={"ID":"2ae34c41-6803-48b8-b0af-4e21bda88175","Type":"ContainerDied","Data":"d2fcd388846e1c5d1a2edfd53f9b032e79d77c3f5ae7a2550828fefceca12d6e"} Oct 30 18:16:54.270697 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.270641 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2fcd388846e1c5d1a2edfd53f9b032e79d77c3f5ae7a2550828fefceca12d6e" Oct 30 18:16:54.270697 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.270490 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs" Oct 30 18:16:54.302358 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.302261 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmsfp\" (UniqueName: \"kubernetes.io/projected/5b6de13a-057e-4364-bded-d2c1a971b9a4-kube-api-access-cmsfp\") pod \"maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h\" (UID: \"5b6de13a-057e-4364-bded-d2c1a971b9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:54.302358 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.302303 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h\" (UID: \"5b6de13a-057e-4364-bded-d2c1a971b9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:54.302563 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.302361 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h\" (UID: \"5b6de13a-057e-4364-bded-d2c1a971b9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:54.403979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.403914 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmsfp\" (UniqueName: \"kubernetes.io/projected/5b6de13a-057e-4364-bded-d2c1a971b9a4-kube-api-access-cmsfp\") pod \"maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h\" (UID: \"5b6de13a-057e-4364-bded-d2c1a971b9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:54.403979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.403993 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h\" (UID: \"5b6de13a-057e-4364-bded-d2c1a971b9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:54.404254 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.404058 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h\" (UID: \"5b6de13a-057e-4364-bded-d2c1a971b9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:54.404801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.404776 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h\" (UID: \"5b6de13a-057e-4364-bded-d2c1a971b9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:54.406887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.406868 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h\" (UID: \"5b6de13a-057e-4364-bded-d2c1a971b9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:54.431310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.431273 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmsfp\" (UniqueName: \"kubernetes.io/projected/5b6de13a-057e-4364-bded-d2c1a971b9a4-kube-api-access-cmsfp\") pod \"maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h\" (UID: \"5b6de13a-057e-4364-bded-d2c1a971b9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:54.559004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.558863 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:54.871246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:54.871215 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h"] Oct 30 18:16:54.872384 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:16:54.872357 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6de13a_057e_4364_bded_d2c1a971b9a4.slice/crio-5c2bf6a6cc627e311a0dabeda225cbacbef16b908f7c9b023309d575b53ffa14 WatchSource:0}: Error finding container 5c2bf6a6cc627e311a0dabeda225cbacbef16b908f7c9b023309d575b53ffa14: Status 404 returned error can't find the container with id 5c2bf6a6cc627e311a0dabeda225cbacbef16b908f7c9b023309d575b53ffa14 Oct 30 18:16:55.178468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:55.178386 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3" path="/var/lib/kubelet/pods/3d3a8aae-4aa1-44d5-ad5a-9d80035bfbf3/volumes" Oct 30 18:16:55.279557 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:55.279526 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" event={"ID":"5b6de13a-057e-4364-bded-d2c1a971b9a4","Type":"ContainerStarted","Data":"5c2bf6a6cc627e311a0dabeda225cbacbef16b908f7c9b023309d575b53ffa14"} Oct 30 18:16:56.303315 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:56.303203 4883 generic.go:356] "Generic (PLEG): container finished" podID="5b6de13a-057e-4364-bded-d2c1a971b9a4" containerID="ec10e7c8d9c8f6b93a2add458771fa11e7f81c95f3898217bbf40569d3d6411a" exitCode=0 Oct 30 18:16:56.303747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:56.303462 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" event={"ID":"5b6de13a-057e-4364-bded-d2c1a971b9a4","Type":"ContainerDied","Data":"ec10e7c8d9c8f6b93a2add458771fa11e7f81c95f3898217bbf40569d3d6411a"} Oct 30 18:16:57.479011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:57.478977 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:57.549556 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:57.549512 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-xks-ca\") pod \"5b6de13a-057e-4364-bded-d2c1a971b9a4\" (UID: \"5b6de13a-057e-4364-bded-d2c1a971b9a4\") " Oct 30 18:16:57.549812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:57.549794 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6de13a-057e-4364-bded-d2c1a971b9a4-kube-api-access-cmsfp\" (UniqueName: \"kubernetes.io/projected/5b6de13a-057e-4364-bded-d2c1a971b9a4-kube-api-access-cmsfp\") pod \"5b6de13a-057e-4364-bded-d2c1a971b9a4\" (UID: \"5b6de13a-057e-4364-bded-d2c1a971b9a4\") " Oct 30 18:16:57.550017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:57.550000 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-openshift-ca\") pod \"5b6de13a-057e-4364-bded-d2c1a971b9a4\" (UID: \"5b6de13a-057e-4364-bded-d2c1a971b9a4\") " Oct 30 18:16:57.550619 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:57.550588 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-openshift-ca" pod "5b6de13a-057e-4364-bded-d2c1a971b9a4" (UID: "5b6de13a-057e-4364-bded-d2c1a971b9a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:57.551112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:57.551087 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:57.553110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:57.553074 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-xks-ca" pod "5b6de13a-057e-4364-bded-d2c1a971b9a4" (UID: "5b6de13a-057e-4364-bded-d2c1a971b9a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:57.553110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:57.553096 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6de13a-057e-4364-bded-d2c1a971b9a4-kube-api-access-cmsfp" pod "5b6de13a-057e-4364-bded-d2c1a971b9a4" (UID: "5b6de13a-057e-4364-bded-d2c1a971b9a4"). InnerVolumeSpecName "kube-api-access-cmsfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:57.652404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:57.652361 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b6de13a-057e-4364-bded-d2c1a971b9a4-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:57.652404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:57.652400 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmsfp\" (UniqueName: \"kubernetes.io/projected/5b6de13a-057e-4364-bded-d2c1a971b9a4-kube-api-access-cmsfp\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:16:58.321901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:58.321867 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" Oct 30 18:16:58.321901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:58.321895 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h" event={"ID":"5b6de13a-057e-4364-bded-d2c1a971b9a4","Type":"ContainerDied","Data":"5c2bf6a6cc627e311a0dabeda225cbacbef16b908f7c9b023309d575b53ffa14"} Oct 30 18:16:58.322153 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:16:58.321960 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c2bf6a6cc627e311a0dabeda225cbacbef16b908f7c9b023309d575b53ffa14" Oct 30 18:17:05.086921 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:17:05.086829 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:17:05.830423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.830383 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9"] Oct 30 18:17:05.831039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.831021 4883 cpu_manager.go:414] "containerMap: removing container" podUID="5b6de13a-057e-4364-bded-d2c1a971b9a4" containerName="revoke-keys" Oct 30 18:17:05.831039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.831040 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6de13a-057e-4364-bded-d2c1a971b9a4" containerName="revoke-keys" Oct 30 18:17:05.831148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.831139 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6de13a-057e-4364-bded-d2c1a971b9a4" containerName="revoke-keys" Oct 30 18:17:05.834309 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.834289 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:05.864478 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.864434 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:17:05.864732 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.864447 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:17:05.864732 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.864443 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:17:05.864948 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.864515 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:17:05.870455 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.870418 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9"] Oct 30 18:17:05.973232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.973181 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77227337-d235-4baf-8b11-4bf73c735420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c048b0b-644j9\" (UID: \"77227337-d235-4baf-8b11-4bf73c735420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:05.973444 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.973260 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f928\" (UniqueName: \"kubernetes.io/projected/77227337-d235-4baf-8b11-4bf73c735420-kube-api-access-2f928\") pod \"maas-api-revoke-sub-5c048b0b-644j9\" (UID: \"77227337-d235-4baf-8b11-4bf73c735420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:05.973621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:05.973587 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77227337-d235-4baf-8b11-4bf73c735420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c048b0b-644j9\" (UID: \"77227337-d235-4baf-8b11-4bf73c735420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:06.075249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:06.075208 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77227337-d235-4baf-8b11-4bf73c735420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c048b0b-644j9\" (UID: \"77227337-d235-4baf-8b11-4bf73c735420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:06.075537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:06.075384 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77227337-d235-4baf-8b11-4bf73c735420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c048b0b-644j9\" (UID: \"77227337-d235-4baf-8b11-4bf73c735420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:06.075537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:06.075425 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f928\" (UniqueName: \"kubernetes.io/projected/77227337-d235-4baf-8b11-4bf73c735420-kube-api-access-2f928\") pod \"maas-api-revoke-sub-5c048b0b-644j9\" (UID: \"77227337-d235-4baf-8b11-4bf73c735420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:06.076279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:06.076251 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77227337-d235-4baf-8b11-4bf73c735420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c048b0b-644j9\" (UID: \"77227337-d235-4baf-8b11-4bf73c735420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:06.078185 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:06.078153 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77227337-d235-4baf-8b11-4bf73c735420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c048b0b-644j9\" (UID: \"77227337-d235-4baf-8b11-4bf73c735420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:06.100106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:06.100066 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f928\" (UniqueName: \"kubernetes.io/projected/77227337-d235-4baf-8b11-4bf73c735420-kube-api-access-2f928\") pod \"maas-api-revoke-sub-5c048b0b-644j9\" (UID: \"77227337-d235-4baf-8b11-4bf73c735420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:06.150182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:06.150143 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:06.462427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:06.462394 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9"] Oct 30 18:17:06.464033 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:17:06.463919 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77227337_d235_4baf_8b11_4bf73c735420.slice/crio-d549fa256e00df59d3f2f68f1e6a001e626599daa6f2aeb36d4a052952f85236 WatchSource:0}: Error finding container d549fa256e00df59d3f2f68f1e6a001e626599daa6f2aeb36d4a052952f85236: Status 404 returned error can't find the container with id d549fa256e00df59d3f2f68f1e6a001e626599daa6f2aeb36d4a052952f85236 Oct 30 18:17:07.400926 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:07.400885 4883 generic.go:356] "Generic (PLEG): container finished" podID="77227337-d235-4baf-8b11-4bf73c735420" containerID="d067498ae874f2ae3bbb607c512a09826f6b9e9cbc419577a8ad42631bdfc405" exitCode=0 Oct 30 18:17:07.401349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:07.400986 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" event={"ID":"77227337-d235-4baf-8b11-4bf73c735420","Type":"ContainerDied","Data":"d067498ae874f2ae3bbb607c512a09826f6b9e9cbc419577a8ad42631bdfc405"} Oct 30 18:17:07.401349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:07.401057 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" event={"ID":"77227337-d235-4baf-8b11-4bf73c735420","Type":"ContainerStarted","Data":"d549fa256e00df59d3f2f68f1e6a001e626599daa6f2aeb36d4a052952f85236"} Oct 30 18:17:08.573683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:08.573650 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:08.726002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:08.725893 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77227337-d235-4baf-8b11-4bf73c735420-kube-api-access-2f928\" (UniqueName: \"kubernetes.io/projected/77227337-d235-4baf-8b11-4bf73c735420-kube-api-access-2f928\") pod \"77227337-d235-4baf-8b11-4bf73c735420\" (UID: \"77227337-d235-4baf-8b11-4bf73c735420\") " Oct 30 18:17:08.726002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:08.725962 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77227337-d235-4baf-8b11-4bf73c735420-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77227337-d235-4baf-8b11-4bf73c735420-maas-api-xks-ca\") pod \"77227337-d235-4baf-8b11-4bf73c735420\" (UID: \"77227337-d235-4baf-8b11-4bf73c735420\") " Oct 30 18:17:08.726259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:08.726176 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77227337-d235-4baf-8b11-4bf73c735420-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77227337-d235-4baf-8b11-4bf73c735420-maas-api-openshift-ca\") pod \"77227337-d235-4baf-8b11-4bf73c735420\" (UID: \"77227337-d235-4baf-8b11-4bf73c735420\") " Oct 30 18:17:08.726500 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:08.726470 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77227337-d235-4baf-8b11-4bf73c735420-maas-api-openshift-ca" pod "77227337-d235-4baf-8b11-4bf73c735420" (UID: "77227337-d235-4baf-8b11-4bf73c735420"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:08.726671 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:08.726652 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77227337-d235-4baf-8b11-4bf73c735420-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:17:08.728226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:08.728203 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77227337-d235-4baf-8b11-4bf73c735420-kube-api-access-2f928" pod "77227337-d235-4baf-8b11-4bf73c735420" (UID: "77227337-d235-4baf-8b11-4bf73c735420"). InnerVolumeSpecName "kube-api-access-2f928". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:08.728308 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:08.728255 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77227337-d235-4baf-8b11-4bf73c735420-maas-api-xks-ca" pod "77227337-d235-4baf-8b11-4bf73c735420" (UID: "77227337-d235-4baf-8b11-4bf73c735420"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:08.828130 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:08.828096 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f928\" (UniqueName: \"kubernetes.io/projected/77227337-d235-4baf-8b11-4bf73c735420-kube-api-access-2f928\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:17:08.828130 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:08.828130 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77227337-d235-4baf-8b11-4bf73c735420-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:17:09.424258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:09.424227 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" Oct 30 18:17:09.425252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:09.424228 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9" event={"ID":"77227337-d235-4baf-8b11-4bf73c735420","Type":"ContainerDied","Data":"d549fa256e00df59d3f2f68f1e6a001e626599daa6f2aeb36d4a052952f85236"} Oct 30 18:17:09.425451 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:09.425280 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d549fa256e00df59d3f2f68f1e6a001e626599daa6f2aeb36d4a052952f85236" Oct 30 18:17:12.399958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:12.392455 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:17:12.533962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:12.533052 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:12.963967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:12.963035 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:17:12.982961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:12.980196 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:12.982961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:12.980643 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:17:13.125131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.125072 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:13.141086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.141036 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:17:13.179962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.176023 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:17:13.322157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.322041 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:13.322816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.322775 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:17:13.328967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.324434 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:17:13.398089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.398037 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:13.398735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.398704 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:17:13.399567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.399544 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:17:13.404835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.404022 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:13.599084 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.599040 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:17:13.616382 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.616339 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:17:13.971946 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.971821 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:13.973069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.973041 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:17:14.081961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:13.999962 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:17:14.081961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.000002 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:17:14.081961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.071134 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:17:14.106960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.082724 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:17:14.243028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.139698 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:14.294049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.150242 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:17:14.294049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.241089 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:14.294049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.250711 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:17:14.294049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.250772 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:17:14.325654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.325610 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:14.326385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.326335 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:17:14.338978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.330081 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:17:14.338978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.330157 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:17:14.338978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.330597 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:14.338978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.331145 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:17:14.373988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.371435 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:17:14.399993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.399419 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:17:14.399993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.399876 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:17:14.438970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.405019 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:14.438970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.435568 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:17:14.438970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.435660 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:17:14.609179 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:14.609134 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:15.309958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.306148 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:17:15.331816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.331768 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:17:15.366470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.342088 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:17:15.383215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.383161 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:15.434882 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.434836 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:17:15.446135 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.446082 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:17:15.495172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.495113 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:15.527706 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.527651 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:17:15.553048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.524419 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:15.646680 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.646636 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:17:15.813003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.648366 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:17:15.813003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.650058 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:15.813003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.572750 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:17:15.813003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:15.672970 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.380100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.380042 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:17:16.380997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.380960 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.403181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.403138 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.424992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.419213 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.484683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.484636 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:17:16.488789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.488745 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:17:16.547034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.537062 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.671879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.671755 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.672129 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.672102 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.673006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.672974 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:17:16.804958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.804120 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:17:16.805147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.805116 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.808959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.805395 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:16.808959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.805620 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:17:16.891959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:16.891383 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded" Oct 30 18:17:17.048039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.047228 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9"] Oct 30 18:17:17.140822 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.140775 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b5160f3-2bmh9"] Oct 30 18:17:17.221050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.221007 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="973674e3-e3f7-4193-8649-2ba7854458b8" path="/var/lib/kubelet/pods/973674e3-e3f7-4193-8649-2ba7854458b8/volumes" Oct 30 18:17:17.390964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.387049 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.390964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.387487 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:17:17.437961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.437436 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.437961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.437763 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:17:17.492093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.491044 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.542076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.541054 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:17:17.549137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.549067 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.675232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.674043 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.690648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.690043 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.805087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.805036 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:17:17.817978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.805795 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:17.817978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.806092 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:17:17.817978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:17.805374 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.187103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.187041 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.388160 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.388100 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:17:18.388898 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.388243 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:17:18.440162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.440043 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.443960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.440462 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:17:18.443960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.442228 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded" Oct 30 18:17:18.499136 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.499085 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.554237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.554190 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:17:18.571960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.556610 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.687961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.683336 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:17:18.715035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.706454 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.830963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.807713 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:17:18.831591 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.808704 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.838960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.808757 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:18.864960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:18.808777 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.030081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.029982 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.142016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.141981 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:17:19.251717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.249540 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:17:19.329293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.329206 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:17:19.346243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.346205 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.347550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.347525 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:17:19.370122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.370080 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:17:19.418966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.409272 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.467246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.467213 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:17:19.467518 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.467500 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:17:19.468891 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.468855 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:19.475959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.472689 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:17:19.475959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.473605 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:17:19.475959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.475279 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:17:19.475959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.475739 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:17:19.488841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.488806 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:17:19.492951 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:19.492892 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:17:21.422828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:21.422793 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:17:21.429262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:21.429229 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:17:21.429649 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:21.429625 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:17:21.436855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:21.436830 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:17:21.864135 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:21.864102 4883 scope.go:122] "RemoveContainer" containerID="ccf603365b7b48c6d7b4f8c7d9e60565a794d0165605ba8624c3ce98e418bb06" Oct 30 18:17:21.874668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:21.874644 4883 scope.go:122] "RemoveContainer" containerID="b47722d1ffca4f38345756cce58dfe58b2ac59b13f5dd8795f8b70b084c74710" Oct 30 18:17:21.888523 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:21.888498 4883 scope.go:122] "RemoveContainer" containerID="e4733297d180f132ceb715f4f598fde4b4e6f4a962393eb64a9a7a0ca3a735b3" Oct 30 18:17:21.899209 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:21.899166 4883 scope.go:122] "RemoveContainer" containerID="fc40075c0672c969b876215668a350cc5fb2bf4de34663b1323a03b1ce152604" Oct 30 18:17:21.910492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:21.910469 4883 scope.go:122] "RemoveContainer" containerID="b9f1515a31b6f6b7c25e0eac6cf4bbd72a2c84c54b53ea0b5cb77241eb68ff01" Oct 30 18:17:21.920754 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:21.920729 4883 scope.go:122] "RemoveContainer" containerID="cb3a9b34912a40b2d5953515c0ad746cf166178026bfa18800a502c15205036b" Oct 30 18:17:21.930825 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:21.930800 4883 scope.go:122] "RemoveContainer" containerID="ce590409d8d7286a892ae3a10e714a2793cc9cd577158ea26f3f3978c0cda58a" Oct 30 18:17:21.941603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:21.941579 4883 scope.go:122] "RemoveContainer" containerID="235e0079332b6288c51c97051ae2703aaba0398b1ca9c8c0aeccdb82c4fbc843" Oct 30 18:17:28.473252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:28.473213 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs"] Oct 30 18:17:28.490021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:28.486633 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-4jbvs"] Oct 30 18:17:28.997327 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:28.997287 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h"] Oct 30 18:17:29.009531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:29.009497 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8f010-vkh5h"] Oct 30 18:17:29.180391 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:29.180355 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae34c41-6803-48b8-b0af-4e21bda88175" path="/var/lib/kubelet/pods/2ae34c41-6803-48b8-b0af-4e21bda88175/volumes" Oct 30 18:17:29.180848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:29.180834 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6de13a-057e-4364-bded-d2c1a971b9a4" path="/var/lib/kubelet/pods/5b6de13a-057e-4364-bded-d2c1a971b9a4/volumes" Oct 30 18:17:35.089671 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:17:35.089621 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:17:35.756389 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.756348 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv"] Oct 30 18:17:35.757131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.757109 4883 cpu_manager.go:414] "containerMap: removing container" podUID="77227337-d235-4baf-8b11-4bf73c735420" containerName="revoke-keys" Oct 30 18:17:35.757131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.757131 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77227337-d235-4baf-8b11-4bf73c735420" containerName="revoke-keys" Oct 30 18:17:35.757308 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.757261 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="77227337-d235-4baf-8b11-4bf73c735420" containerName="revoke-keys" Oct 30 18:17:35.761052 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.761023 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:35.782753 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.782721 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv"] Oct 30 18:17:35.794997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.790831 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:17:35.794997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.791032 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:17:35.799754 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.799724 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:17:35.800053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.799878 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:17:35.875580 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.875539 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2769dcb0-26ptv\" (UID: \"5d42db29-a73b-47ec-aa49-afadad615cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:35.875757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.875703 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2769dcb0-26ptv\" (UID: \"5d42db29-a73b-47ec-aa49-afadad615cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:35.875885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.875870 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc675\" (UniqueName: \"kubernetes.io/projected/5d42db29-a73b-47ec-aa49-afadad615cfa-kube-api-access-hc675\") pod \"maas-api-revoke-sub-2769dcb0-26ptv\" (UID: \"5d42db29-a73b-47ec-aa49-afadad615cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:35.977133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.977092 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc675\" (UniqueName: \"kubernetes.io/projected/5d42db29-a73b-47ec-aa49-afadad615cfa-kube-api-access-hc675\") pod \"maas-api-revoke-sub-2769dcb0-26ptv\" (UID: \"5d42db29-a73b-47ec-aa49-afadad615cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:35.977327 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.977169 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2769dcb0-26ptv\" (UID: \"5d42db29-a73b-47ec-aa49-afadad615cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:35.977327 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.977248 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2769dcb0-26ptv\" (UID: \"5d42db29-a73b-47ec-aa49-afadad615cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:35.977856 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.977831 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2769dcb0-26ptv\" (UID: \"5d42db29-a73b-47ec-aa49-afadad615cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:35.979740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:35.979706 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2769dcb0-26ptv\" (UID: \"5d42db29-a73b-47ec-aa49-afadad615cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:36.002164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:36.002125 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc675\" (UniqueName: \"kubernetes.io/projected/5d42db29-a73b-47ec-aa49-afadad615cfa-kube-api-access-hc675\") pod \"maas-api-revoke-sub-2769dcb0-26ptv\" (UID: \"5d42db29-a73b-47ec-aa49-afadad615cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:36.075785 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:36.075697 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:36.389976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:36.385613 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv"] Oct 30 18:17:36.392865 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:17:36.392836 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d42db29_a73b_47ec_aa49_afadad615cfa.slice/crio-45617d2543602f9d1007b434899a0c574cdddd15adcde768eca5a8a2e6936068 WatchSource:0}: Error finding container 45617d2543602f9d1007b434899a0c574cdddd15adcde768eca5a8a2e6936068: Status 404 returned error can't find the container with id 45617d2543602f9d1007b434899a0c574cdddd15adcde768eca5a8a2e6936068 Oct 30 18:17:36.968582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:36.968536 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" event={"ID":"5d42db29-a73b-47ec-aa49-afadad615cfa","Type":"ContainerStarted","Data":"1f68c50851b48b2b71ee4f4a3120ff108151a04410b82330ae329bfbdbcb2284"} Oct 30 18:17:36.968822 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:36.968608 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" event={"ID":"5d42db29-a73b-47ec-aa49-afadad615cfa","Type":"ContainerStarted","Data":"45617d2543602f9d1007b434899a0c574cdddd15adcde768eca5a8a2e6936068"} Oct 30 18:17:37.020958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:37.017579 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" podStartSLOduration=2.017550286 podStartE2EDuration="2.017550286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:37.012325943 +0000 UTC m=+1515.969329778" watchObservedRunningTime="2026-10-30 18:17:37.017550286 +0000 UTC m=+1515.974554131" Oct 30 18:17:37.977657 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:37.977620 4883 generic.go:356] "Generic (PLEG): container finished" podID="5d42db29-a73b-47ec-aa49-afadad615cfa" containerID="1f68c50851b48b2b71ee4f4a3120ff108151a04410b82330ae329bfbdbcb2284" exitCode=0 Oct 30 18:17:37.978270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:37.977702 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" event={"ID":"5d42db29-a73b-47ec-aa49-afadad615cfa","Type":"ContainerDied","Data":"1f68c50851b48b2b71ee4f4a3120ff108151a04410b82330ae329bfbdbcb2284"} Oct 30 18:17:38.073893 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.073857 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6"] Oct 30 18:17:38.077841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.077813 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:17:38.101010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.100967 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6"] Oct 30 18:17:38.243142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.243040 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da151d-642j6\" (UID: \"79c3e635-7780-46df-b6e1-455cbce6fec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:17:38.243142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.243085 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l4fh\" (UniqueName: \"kubernetes.io/projected/79c3e635-7780-46df-b6e1-455cbce6fec7-kube-api-access-6l4fh\") pod \"maas-api-revoke-keys-e2e-api-b-da151d-642j6\" (UID: \"79c3e635-7780-46df-b6e1-455cbce6fec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:17:38.243397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.243202 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da151d-642j6\" (UID: \"79c3e635-7780-46df-b6e1-455cbce6fec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:17:38.344527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.344492 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da151d-642j6\" (UID: \"79c3e635-7780-46df-b6e1-455cbce6fec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:17:38.344527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.344534 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l4fh\" (UniqueName: \"kubernetes.io/projected/79c3e635-7780-46df-b6e1-455cbce6fec7-kube-api-access-6l4fh\") pod \"maas-api-revoke-keys-e2e-api-b-da151d-642j6\" (UID: \"79c3e635-7780-46df-b6e1-455cbce6fec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:17:38.344799 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.344577 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da151d-642j6\" (UID: \"79c3e635-7780-46df-b6e1-455cbce6fec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:17:38.345390 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.345366 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da151d-642j6\" (UID: \"79c3e635-7780-46df-b6e1-455cbce6fec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:17:38.347195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.347174 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da151d-642j6\" (UID: \"79c3e635-7780-46df-b6e1-455cbce6fec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:17:38.369594 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.369560 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l4fh\" (UniqueName: \"kubernetes.io/projected/79c3e635-7780-46df-b6e1-455cbce6fec7-kube-api-access-6l4fh\") pod \"maas-api-revoke-keys-e2e-api-b-da151d-642j6\" (UID: \"79c3e635-7780-46df-b6e1-455cbce6fec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:17:38.393559 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.393525 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:17:38.724603 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:17:38.724565 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c3e635_7780_46df_b6e1_455cbce6fec7.slice/crio-3b346c0cba13dbfb82824eac68a4ab62014e6c363f6802d4f7c3ddb62b58b6dc WatchSource:0}: Error finding container 3b346c0cba13dbfb82824eac68a4ab62014e6c363f6802d4f7c3ddb62b58b6dc: Status 404 returned error can't find the container with id 3b346c0cba13dbfb82824eac68a4ab62014e6c363f6802d4f7c3ddb62b58b6dc Oct 30 18:17:38.724756 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:38.724594 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6"] Oct 30 18:17:39.009918 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:39.009877 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" event={"ID":"79c3e635-7780-46df-b6e1-455cbce6fec7","Type":"ContainerStarted","Data":"3b346c0cba13dbfb82824eac68a4ab62014e6c363f6802d4f7c3ddb62b58b6dc"} Oct 30 18:17:39.162642 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:39.162610 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:39.182399 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:39.182371 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d42db29-a73b-47ec-aa49-afadad615cfa-kube-api-access-hc675\" (UniqueName: \"kubernetes.io/projected/5d42db29-a73b-47ec-aa49-afadad615cfa-kube-api-access-hc675\") pod \"5d42db29-a73b-47ec-aa49-afadad615cfa\" (UID: \"5d42db29-a73b-47ec-aa49-afadad615cfa\") " Oct 30 18:17:39.182570 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:39.182558 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-xks-ca\") pod \"5d42db29-a73b-47ec-aa49-afadad615cfa\" (UID: \"5d42db29-a73b-47ec-aa49-afadad615cfa\") " Oct 30 18:17:39.182637 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:39.182620 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-openshift-ca\") pod \"5d42db29-a73b-47ec-aa49-afadad615cfa\" (UID: \"5d42db29-a73b-47ec-aa49-afadad615cfa\") " Oct 30 18:17:39.182984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:39.182959 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-openshift-ca" pod "5d42db29-a73b-47ec-aa49-afadad615cfa" (UID: "5d42db29-a73b-47ec-aa49-afadad615cfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:39.183093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:39.183080 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:17:39.184503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:39.184474 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-xks-ca" pod "5d42db29-a73b-47ec-aa49-afadad615cfa" (UID: "5d42db29-a73b-47ec-aa49-afadad615cfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:39.184595 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:39.184507 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d42db29-a73b-47ec-aa49-afadad615cfa-kube-api-access-hc675" pod "5d42db29-a73b-47ec-aa49-afadad615cfa" (UID: "5d42db29-a73b-47ec-aa49-afadad615cfa"). InnerVolumeSpecName "kube-api-access-hc675". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:39.284485 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:39.284395 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d42db29-a73b-47ec-aa49-afadad615cfa-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:17:39.284485 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:39.284432 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc675\" (UniqueName: \"kubernetes.io/projected/5d42db29-a73b-47ec-aa49-afadad615cfa-kube-api-access-hc675\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:17:40.022993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:40.022293 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" Oct 30 18:17:40.022993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:40.022323 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv" event={"ID":"5d42db29-a73b-47ec-aa49-afadad615cfa","Type":"ContainerDied","Data":"45617d2543602f9d1007b434899a0c574cdddd15adcde768eca5a8a2e6936068"} Oct 30 18:17:40.022993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:40.022408 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45617d2543602f9d1007b434899a0c574cdddd15adcde768eca5a8a2e6936068" Oct 30 18:17:40.025987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:40.024593 4883 generic.go:356] "Generic (PLEG): container finished" podID="79c3e635-7780-46df-b6e1-455cbce6fec7" containerID="9b5d5629c4d182ae445863c458bd60bec877444b8588294440c94b72d965e7aa" exitCode=0 Oct 30 18:17:40.025987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:40.024644 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" event={"ID":"79c3e635-7780-46df-b6e1-455cbce6fec7","Type":"ContainerDied","Data":"9b5d5629c4d182ae445863c458bd60bec877444b8588294440c94b72d965e7aa"} Oct 30 18:17:41.212231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:41.212202 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:17:41.329329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:41.329235 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-openshift-ca\") pod \"79c3e635-7780-46df-b6e1-455cbce6fec7\" (UID: \"79c3e635-7780-46df-b6e1-455cbce6fec7\") " Oct 30 18:17:41.329492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:41.329467 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-xks-ca\") pod \"79c3e635-7780-46df-b6e1-455cbce6fec7\" (UID: \"79c3e635-7780-46df-b6e1-455cbce6fec7\") " Oct 30 18:17:41.329536 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:41.329518 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c3e635-7780-46df-b6e1-455cbce6fec7-kube-api-access-6l4fh\" (UniqueName: \"kubernetes.io/projected/79c3e635-7780-46df-b6e1-455cbce6fec7-kube-api-access-6l4fh\") pod \"79c3e635-7780-46df-b6e1-455cbce6fec7\" (UID: \"79c3e635-7780-46df-b6e1-455cbce6fec7\") " Oct 30 18:17:41.329713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:41.329672 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-openshift-ca" pod "79c3e635-7780-46df-b6e1-455cbce6fec7" (UID: "79c3e635-7780-46df-b6e1-455cbce6fec7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:41.330117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:41.330098 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:17:41.331691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:41.331663 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-xks-ca" pod "79c3e635-7780-46df-b6e1-455cbce6fec7" (UID: "79c3e635-7780-46df-b6e1-455cbce6fec7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:41.331780 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:41.331718 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c3e635-7780-46df-b6e1-455cbce6fec7-kube-api-access-6l4fh" pod "79c3e635-7780-46df-b6e1-455cbce6fec7" (UID: "79c3e635-7780-46df-b6e1-455cbce6fec7"). InnerVolumeSpecName "kube-api-access-6l4fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:41.431491 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:41.431446 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c3e635-7780-46df-b6e1-455cbce6fec7-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:17:41.431491 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:41.431484 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l4fh\" (UniqueName: \"kubernetes.io/projected/79c3e635-7780-46df-b6e1-455cbce6fec7-kube-api-access-6l4fh\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:17:42.130009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:42.127955 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" event={"ID":"79c3e635-7780-46df-b6e1-455cbce6fec7","Type":"ContainerDied","Data":"3b346c0cba13dbfb82824eac68a4ab62014e6c363f6802d4f7c3ddb62b58b6dc"} Oct 30 18:17:42.130009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:42.128025 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b346c0cba13dbfb82824eac68a4ab62014e6c363f6802d4f7c3ddb62b58b6dc" Oct 30 18:17:42.130009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:17:42.128099 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6" Oct 30 18:18:05.081186 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:18:05.081147 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:18:06.757061 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.757015 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7"] Oct 30 18:18:06.757668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.757649 4883 cpu_manager.go:414] "containerMap: removing container" podUID="79c3e635-7780-46df-b6e1-455cbce6fec7" containerName="revoke-keys" Oct 30 18:18:06.757725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.757670 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c3e635-7780-46df-b6e1-455cbce6fec7" containerName="revoke-keys" Oct 30 18:18:06.757725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.757690 4883 cpu_manager.go:414] "containerMap: removing container" podUID="5d42db29-a73b-47ec-aa49-afadad615cfa" containerName="revoke-keys" Oct 30 18:18:06.757725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.757695 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d42db29-a73b-47ec-aa49-afadad615cfa" containerName="revoke-keys" Oct 30 18:18:06.757886 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.757807 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c3e635-7780-46df-b6e1-455cbce6fec7" containerName="revoke-keys" Oct 30 18:18:06.757886 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.757820 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d42db29-a73b-47ec-aa49-afadad615cfa" containerName="revoke-keys" Oct 30 18:18:06.761878 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.761848 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:06.778997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.778665 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:18:06.779210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.778677 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:18:06.779210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.778696 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:18:06.779210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.778762 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:18:06.781220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.781192 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7"] Oct 30 18:18:06.858641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.858607 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7\" (UID: \"9d529571-e86c-4289-b6c0-1a92922a7e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:06.858819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.858794 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwh6v\" (UniqueName: \"kubernetes.io/projected/9d529571-e86c-4289-b6c0-1a92922a7e66-kube-api-access-vwh6v\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7\" (UID: \"9d529571-e86c-4289-b6c0-1a92922a7e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:06.859010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.858972 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7\" (UID: \"9d529571-e86c-4289-b6c0-1a92922a7e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:06.960363 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.960322 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwh6v\" (UniqueName: \"kubernetes.io/projected/9d529571-e86c-4289-b6c0-1a92922a7e66-kube-api-access-vwh6v\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7\" (UID: \"9d529571-e86c-4289-b6c0-1a92922a7e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:06.960549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.960515 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7\" (UID: \"9d529571-e86c-4289-b6c0-1a92922a7e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:06.960620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.960604 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7\" (UID: \"9d529571-e86c-4289-b6c0-1a92922a7e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:06.961236 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.961207 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7\" (UID: \"9d529571-e86c-4289-b6c0-1a92922a7e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:06.963200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.963171 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7\" (UID: \"9d529571-e86c-4289-b6c0-1a92922a7e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:06.985861 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:06.985358 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwh6v\" (UniqueName: \"kubernetes.io/projected/9d529571-e86c-4289-b6c0-1a92922a7e66-kube-api-access-vwh6v\") pod \"maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7\" (UID: \"9d529571-e86c-4289-b6c0-1a92922a7e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:07.080405 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:07.078267 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:07.460272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:07.460231 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7"] Oct 30 18:18:08.358629 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:08.358588 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" event={"ID":"9d529571-e86c-4289-b6c0-1a92922a7e66","Type":"ContainerStarted","Data":"b25631f09afad19b094f61800fd9f407eafaf5bdbef1deee1ae02bef625b21c0"} Oct 30 18:18:08.359074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:08.358669 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" event={"ID":"9d529571-e86c-4289-b6c0-1a92922a7e66","Type":"ContainerStarted","Data":"b60908701c726649e326767d242561018d9332f26b4cf0cb4a8fa87ca56b97a5"} Oct 30 18:18:08.408484 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:08.408417 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" podStartSLOduration=2.40839781 podStartE2EDuration="2.40839781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:08.403048169 +0000 UTC m=+1547.360052006" watchObservedRunningTime="2026-10-30 18:18:08.40839781 +0000 UTC m=+1547.365401647" Oct 30 18:18:09.370789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:09.370749 4883 generic.go:356] "Generic (PLEG): container finished" podID="9d529571-e86c-4289-b6c0-1a92922a7e66" containerID="b25631f09afad19b094f61800fd9f407eafaf5bdbef1deee1ae02bef625b21c0" exitCode=0 Oct 30 18:18:09.371349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:09.370847 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" event={"ID":"9d529571-e86c-4289-b6c0-1a92922a7e66","Type":"ContainerDied","Data":"b25631f09afad19b094f61800fd9f407eafaf5bdbef1deee1ae02bef625b21c0"} Oct 30 18:18:10.617098 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:10.617055 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:10.627440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:10.627397 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-openshift-ca\") pod \"9d529571-e86c-4289-b6c0-1a92922a7e66\" (UID: \"9d529571-e86c-4289-b6c0-1a92922a7e66\") " Oct 30 18:18:10.627626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:10.627456 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d529571-e86c-4289-b6c0-1a92922a7e66-kube-api-access-vwh6v\" (UniqueName: \"kubernetes.io/projected/9d529571-e86c-4289-b6c0-1a92922a7e66-kube-api-access-vwh6v\") pod \"9d529571-e86c-4289-b6c0-1a92922a7e66\" (UID: \"9d529571-e86c-4289-b6c0-1a92922a7e66\") " Oct 30 18:18:10.627626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:10.627616 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-xks-ca\") pod \"9d529571-e86c-4289-b6c0-1a92922a7e66\" (UID: \"9d529571-e86c-4289-b6c0-1a92922a7e66\") " Oct 30 18:18:10.627883 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:10.627855 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-openshift-ca" pod "9d529571-e86c-4289-b6c0-1a92922a7e66" (UID: "9d529571-e86c-4289-b6c0-1a92922a7e66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:10.628558 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:10.628527 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:18:10.630241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:10.630192 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-xks-ca" pod "9d529571-e86c-4289-b6c0-1a92922a7e66" (UID: "9d529571-e86c-4289-b6c0-1a92922a7e66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:10.630387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:10.630354 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d529571-e86c-4289-b6c0-1a92922a7e66-kube-api-access-vwh6v" pod "9d529571-e86c-4289-b6c0-1a92922a7e66" (UID: "9d529571-e86c-4289-b6c0-1a92922a7e66"). InnerVolumeSpecName "kube-api-access-vwh6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:10.729618 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:10.729575 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwh6v\" (UniqueName: \"kubernetes.io/projected/9d529571-e86c-4289-b6c0-1a92922a7e66-kube-api-access-vwh6v\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:18:10.729618 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:10.729613 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d529571-e86c-4289-b6c0-1a92922a7e66-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:18:11.032559 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:11.032453 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb"] Oct 30 18:18:11.036578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:11.036537 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b4b6c2-rljpb"] Oct 30 18:18:11.183838 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:11.183794 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257f59ea-768d-4f76-b70f-e2a460559980" path="/var/lib/kubelet/pods/257f59ea-768d-4f76-b70f-e2a460559980/volumes" Oct 30 18:18:11.390744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:11.390700 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" event={"ID":"9d529571-e86c-4289-b6c0-1a92922a7e66","Type":"ContainerDied","Data":"b60908701c726649e326767d242561018d9332f26b4cf0cb4a8fa87ca56b97a5"} Oct 30 18:18:11.390744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:11.390721 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7" Oct 30 18:18:11.391034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:11.390756 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b60908701c726649e326767d242561018d9332f26b4cf0cb4a8fa87ca56b97a5" Oct 30 18:18:13.065654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:13.065606 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6"] Oct 30 18:18:13.078612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:13.078577 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da151d-642j6"] Oct 30 18:18:13.186157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:13.186093 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c3e635-7780-46df-b6e1-455cbce6fec7" path="/var/lib/kubelet/pods/79c3e635-7780-46df-b6e1-455cbce6fec7/volumes" Oct 30 18:18:22.022971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:22.022034 4883 scope.go:122] "RemoveContainer" containerID="027f59d0944f700d2e5fe10e8d302a5052d2a0bfe97c11082236952fc06ea75d" Oct 30 18:18:22.061560 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:22.061526 4883 scope.go:122] "RemoveContainer" containerID="94b8f0abd3b7c7b3a4920545e65becf6d6452535140b61b3d662a6de12d17d4a" Oct 30 18:18:22.084676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:22.084250 4883 scope.go:122] "RemoveContainer" containerID="a488f419e3fb5fdf9bee333a4f36506b29690182eb763048d37b2d2f107a4668" Oct 30 18:18:22.108449 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:22.107997 4883 scope.go:122] "RemoveContainer" containerID="c498901dc9fdee3892d24efdd6cba07c8bb13dfdf9f99f0b2bd9da5924a25b8e" Oct 30 18:18:22.128382 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:22.128349 4883 scope.go:122] "RemoveContainer" containerID="fe1502a336923aee3155b2bd2965b3acf68e5c3ca89185603ef782b7cdd72f7d" Oct 30 18:18:22.151447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:22.150901 4883 scope.go:122] "RemoveContainer" containerID="548967de468c8408243570da9ba6dd2f8e7d5aae1c90da41ee478f0ccde00f14" Oct 30 18:18:34.471024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:34.470200 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:34.550729 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:34.548423 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:18:34.551046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:34.551016 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7"] Oct 30 18:18:34.581672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:34.581628 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c6b2120f-qtmk7"] Oct 30 18:18:34.686141 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:34.686089 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:34.686893 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:34.686400 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:34.803195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:34.803075 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:34.803830 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:34.803798 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:34.805589 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:34.804220 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.254002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.253959 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d529571-e86c-4289-b6c0-1a92922a7e66" path="/var/lib/kubelet/pods/9d529571-e86c-4289-b6c0-1a92922a7e66/volumes" Oct 30 18:18:35.372120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.372076 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.384130 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.384080 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.391124 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.391065 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.403104 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.403056 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.404378 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.404263 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.477199 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.477143 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:18:35.478572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.478530 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:18:35.506316 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:18:35.506213 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:18:35.567138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.567083 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:18:35.567969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.567919 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:18:35.602957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.584916 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded" Oct 30 18:18:35.688973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.688201 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:35.700414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.700370 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:18:35.729961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.701225 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:18:35.999481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.999451 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:18:35.999726 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.999712 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:18:35.999846 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.831545 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.001761 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.831606 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:18:36.072376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.831053 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.072376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:35.831096 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.072376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.046633 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:18:36.072376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.046703 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:18:36.119720 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.119689 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:18:36.398120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.374988 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.412215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.399158 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:18:36.412215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.402731 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:18:36.412215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.406508 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.412215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.408344 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.412215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.409592 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:18:36.445735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.435831 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:18:36.454977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.448655 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:18:36.454977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.448898 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:18:36.454977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.449282 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:18:36.501982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.486978 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:18:36.597961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.573106 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:18:36.760494 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.760373 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:36.760854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:36.760809 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.108969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.103531 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:18:37.108969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.103821 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:18:37.136996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.130334 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.136996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.131476 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded" Oct 30 18:18:37.213043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.204349 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.213043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.204819 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.376763 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.376601 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.429183 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.420082 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.429183 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.424832 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:18:37.430125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.429990 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:18:37.451336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.451269 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:18:37.538785 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.537831 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.715143 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.715030 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:18:37.731961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.725127 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:37.779488 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:37.779422 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.117122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.105078 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:18:38.154962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.130921 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.154962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.134081 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:38.154962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.134160 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.216104 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.216052 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.244385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.216749 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:18:38.245154 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.217716 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.381150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.381039 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:18:38.421091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.421039 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.426469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.426420 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.432968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.431037 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.432968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.431294 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.459975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.459162 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:18:38.521962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.521372 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:18:38.549963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.542854 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.724163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.724049 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.725064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.725037 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:18:38.726393 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.726364 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:38.784959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:38.781031 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.065157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.065039 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Oct 30 18:18:39.116956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.110414 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.230960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.230536 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.248147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.248100 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.251970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.250038 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.278461 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.278427 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:18:39.392958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.391533 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.467960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.465149 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.467960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.467452 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:18:39.504989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.503041 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.659624 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.659514 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:18:39.779998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.685895 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.791979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.697117 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.811247 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.794549 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:18:39.903996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.889211 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:18:39.903996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.889913 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:39.903996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:39.890106 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:18:40.113986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.110601 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.142983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.132665 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.148975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.148585 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t"] Oct 30 18:18:40.253964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.253109 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.460035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.459007 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.476959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.470047 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.476959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.471397 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.654296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.654228 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.663577 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.663521 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.760227 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.760091 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:18:40.805958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.784168 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 18:18:40.843080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.840339 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:18:40.843080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.841344 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.890146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.890085 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.946272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:40.946229 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.122724 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.122667 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:18:41.140123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.140073 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.140971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.140922 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:18:41.255958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.255465 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.326959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.326553 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.470949 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.470598 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.500561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.470233 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:18:41.501487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.474916 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.763191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.761383 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.835171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.835100 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.862150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.862076 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.862612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.862575 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.904152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.904090 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:18:41.951105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:41.951047 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:18:42.180961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.148597 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:18:42.180961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.173039 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:18:42.233329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.233290 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:18:42.280956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.268757 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.309986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.308255 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:18:42.315966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.313596 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:18:42.337850 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.337397 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.337850 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.337690 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:18:42.508049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.503241 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.508049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.504109 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:18:42.637428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.634894 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:18:42.709031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.704217 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:18:42.787758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.775884 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:18:42.797076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.796677 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:18:42.797261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.797173 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:18:42.805571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.803863 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:18:42.805571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.805142 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:18:42.816668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.810679 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:18:42.816668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.811649 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:18:42.816668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:42.813303 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:18:49.126814 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:49.126735 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x"] Oct 30 18:18:49.140047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:49.140012 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83de5561-ndd8x"] Oct 30 18:18:49.179656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:49.179617 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a11e90-5324-4af4-9555-151467c3c2d4" path="/var/lib/kubelet/pods/34a11e90-5324-4af4-9555-151467c3c2d4/volumes" Oct 30 18:18:57.790032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:57.789969 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" containerID="cri-o://6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35" gracePeriod=43 Oct 30 18:18:57.807784 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:57.807747 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": dial tcp 10.132.0.109:8000: connect: connection refused" Oct 30 18:18:58.139412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.139376 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:18:58.236339 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.236308 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-home\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-home\") pod \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " Oct 30 18:18:58.236548 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.236466 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ec35a1-8960-44ec-9fb3-eb9df9492b92-kube-api-access-fgm22\" (UniqueName: \"kubernetes.io/projected/57ec35a1-8960-44ec-9fb3-eb9df9492b92-kube-api-access-fgm22\") pod \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " Oct 30 18:18:58.236648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.236633 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-dshm\") pod \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " Oct 30 18:18:58.236707 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.236663 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tmp-dir\") pod \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " Oct 30 18:18:58.236707 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.236681 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-model-cache\") pod \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " Oct 30 18:18:58.236816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.236712 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tls-certs\" (UniqueName: \"kubernetes.io/secret/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tls-certs\") pod \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\" (UID: \"57ec35a1-8960-44ec-9fb3-eb9df9492b92\") " Oct 30 18:18:58.237304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.237122 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-home" pod "57ec35a1-8960-44ec-9fb3-eb9df9492b92" (UID: "57ec35a1-8960-44ec-9fb3-eb9df9492b92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:58.237304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.237133 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-model-cache" pod "57ec35a1-8960-44ec-9fb3-eb9df9492b92" (UID: "57ec35a1-8960-44ec-9fb3-eb9df9492b92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:58.237304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.237178 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tmp-dir" pod "57ec35a1-8960-44ec-9fb3-eb9df9492b92" (UID: "57ec35a1-8960-44ec-9fb3-eb9df9492b92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:58.237719 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.237696 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:18:58.237801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.237724 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:18:58.237801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.237736 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:18:58.239677 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.239623 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tls-certs" pod "57ec35a1-8960-44ec-9fb3-eb9df9492b92" (UID: "57ec35a1-8960-44ec-9fb3-eb9df9492b92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:58.239677 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.239645 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-dshm" pod "57ec35a1-8960-44ec-9fb3-eb9df9492b92" (UID: "57ec35a1-8960-44ec-9fb3-eb9df9492b92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:58.240410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.240370 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ec35a1-8960-44ec-9fb3-eb9df9492b92-kube-api-access-fgm22" pod "57ec35a1-8960-44ec-9fb3-eb9df9492b92" (UID: "57ec35a1-8960-44ec-9fb3-eb9df9492b92"). InnerVolumeSpecName "kube-api-access-fgm22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:58.339574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.339536 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ec35a1-8960-44ec-9fb3-eb9df9492b92-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:18:58.339574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.339577 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ec35a1-8960-44ec-9fb3-eb9df9492b92-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:18:58.339816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.339594 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgm22\" (UniqueName: \"kubernetes.io/projected/57ec35a1-8960-44ec-9fb3-eb9df9492b92-kube-api-access-fgm22\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:18:58.493423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.493332 4883 generic.go:356] "Generic (PLEG): container finished" podID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerID="6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35" exitCode=0 Oct 30 18:18:58.493423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.493390 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" Oct 30 18:18:58.493423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.493409 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" event={"ID":"57ec35a1-8960-44ec-9fb3-eb9df9492b92","Type":"ContainerDied","Data":"6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35"} Oct 30 18:18:58.493700 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.493457 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t" event={"ID":"57ec35a1-8960-44ec-9fb3-eb9df9492b92","Type":"ContainerDied","Data":"51050b7975f53963d2e20fd41dcd23c7173dc51b1225d51fdb54b1874918b726"} Oct 30 18:18:58.493700 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.493480 4883 scope.go:122] "RemoveContainer" containerID="6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35" Oct 30 18:18:58.506576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.506546 4883 scope.go:122] "RemoveContainer" containerID="6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35" Oct 30 18:18:58.506993 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:18:58.506958 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35\": container with ID starting with 6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35 not found: ID does not exist" containerID="6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35" Oct 30 18:18:58.507117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.507006 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35"} err="failed to get container status \"6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35\": rpc error: code = NotFound desc = could not find container \"6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35\": container with ID starting with 6f36efc7af69ec0b23137c2121afc526bbf60c5091a1b9198a7c912a8438ac35 not found: ID does not exist" Oct 30 18:18:58.555044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.555007 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t"] Oct 30 18:18:58.572867 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:58.572830 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c6b2120f/e2e-lifecycle-model-c6b2120f-kserve-cc9cb49bf-hr82t"] Oct 30 18:18:59.179954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:18:59.179912 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" path="/var/lib/kubelet/pods/57ec35a1-8960-44ec-9fb3-eb9df9492b92/volumes" Oct 30 18:19:05.082290 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:19:05.082246 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:19:08.149157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.148323 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz"] Oct 30 18:19:08.149655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.149234 4883 cpu_manager.go:414] "containerMap: removing container" podUID="9d529571-e86c-4289-b6c0-1a92922a7e66" containerName="revoke-keys" Oct 30 18:19:08.149655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.149254 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d529571-e86c-4289-b6c0-1a92922a7e66" containerName="revoke-keys" Oct 30 18:19:08.149655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.149299 4883 cpu_manager.go:414] "containerMap: removing container" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" Oct 30 18:19:08.149655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.149306 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" Oct 30 18:19:08.149655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.149421 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d529571-e86c-4289-b6c0-1a92922a7e66" containerName="revoke-keys" Oct 30 18:19:08.149655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.149438 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ec35a1-8960-44ec-9fb3-eb9df9492b92" containerName="main" Oct 30 18:19:08.156911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.156869 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.181970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.180786 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz"] Oct 30 18:19:08.295164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.295114 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.295388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.295259 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.295388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.295296 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.295388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.295325 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.295533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.295487 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.295632 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.295580 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.295714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.295631 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhgt\" (UniqueName: \"kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-kube-api-access-vwhgt\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.295799 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.295781 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.295915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.295842 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.406875 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.406989 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhgt\" (UniqueName: \"kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-kube-api-access-vwhgt\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.407122 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.407174 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.407258 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.407366 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.407401 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.407428 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.407460 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.408397 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-credential-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.409538 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-data\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.409761 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-certs\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.410381 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istiod-ca-cert\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.415561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.410611 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-socket\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.426960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.417197 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-envoy\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.426960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.419634 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-podinfo\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.437541 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.437510 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-token\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.479744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.479708 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhgt\" (UniqueName: \"kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-kube-api-access-vwhgt\") pod \"e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.503961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.501132 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:08.874253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.874211 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz"] Oct 30 18:19:08.882009 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:19:08.881979 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44e1f4a_a663_4903_8a3c_0237dd2c3643.slice/crio-2f801893d1a619de471a3023415822e6512aa72b46531c71d5caf4ed360fa9bb WatchSource:0}: Error finding container 2f801893d1a619de471a3023415822e6512aa72b46531c71d5caf4ed360fa9bb: Status 404 returned error can't find the container with id 2f801893d1a619de471a3023415822e6512aa72b46531c71d5caf4ed360fa9bb Oct 30 18:19:08.885544 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.885492 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:08.885677 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.885591 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:08.885677 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:08.885623 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:09.589614 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:09.589552 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" event={"ID":"e44e1f4a-a663-4903-8a3c-0237dd2c3643","Type":"ContainerStarted","Data":"8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9"} Oct 30 18:19:09.590127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:09.589629 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" event={"ID":"e44e1f4a-a663-4903-8a3c-0237dd2c3643","Type":"ContainerStarted","Data":"2f801893d1a619de471a3023415822e6512aa72b46531c71d5caf4ed360fa9bb"} Oct 30 18:19:09.654741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:09.654678 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" podStartSLOduration=1.654661572 podStartE2EDuration="1.654661572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:09.650953566 +0000 UTC m=+1608.607957397" watchObservedRunningTime="2026-10-30 18:19:09.654661572 +0000 UTC m=+1608.611665407" Oct 30 18:19:10.502770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:10.502732 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:10.504216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:10.504183 4883 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 18:19:10.504393 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:10.504235 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" podUID="e44e1f4a-a663-4903-8a3c-0237dd2c3643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 18:19:11.502509 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:11.502471 4883 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 18:19:11.504982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:11.502526 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" podUID="e44e1f4a-a663-4903-8a3c-0237dd2c3643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 18:19:12.527022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:12.503342 4883 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 18:19:12.527022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:12.503411 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" podUID="e44e1f4a-a663-4903-8a3c-0237dd2c3643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 18:19:13.501858 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:13.501820 4883 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 18:19:13.502059 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:13.501877 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" podUID="e44e1f4a-a663-4903-8a3c-0237dd2c3643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 18:19:14.502651 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:14.502616 4883 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 18:19:14.503123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:14.502674 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" podUID="e44e1f4a-a663-4903-8a3c-0237dd2c3643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 18:19:15.533157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:15.508468 4883 patch_prober.go:29] interesting pod/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 18:19:15.533860 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:15.533814 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" podUID="e44e1f4a-a663-4903-8a3c-0237dd2c3643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 18:19:16.517741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:16.517700 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:16.520013 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:16.519907 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:16.531667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:16.531637 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:22.243256 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:22.243223 4883 scope.go:122] "RemoveContainer" containerID="95a8b6a2b2b0cfc75cd56af332a770a96fecd506a2d6a2fb8682e5023444febd" Oct 30 18:19:24.208900 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:24.208857 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz"] Oct 30 18:19:24.209417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:24.209132 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" podUID="e44e1f4a-a663-4903-8a3c-0237dd2c3643" containerName="istio-proxy" containerID="cri-o://8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9" gracePeriod=30 Oct 30 18:19:24.222929 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:24.222631 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8"] Oct 30 18:19:24.226925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:24.223509 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" podUID="07abf63a-7209-4001-90dc-af2cb30525ec" containerName="istio-proxy" containerID="cri-o://2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a" gracePeriod=30 Oct 30 18:19:25.491961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.491024 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg"] Oct 30 18:19:25.497910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.497662 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.531000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.526155 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg"] Oct 30 18:19:25.531000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.527280 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a3be60-openshift-default-dockercfg-kcfpl\"" Oct 30 18:19:25.567060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.561521 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-envoy\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.567060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.561653 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-credential-socket\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.567060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.561685 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istiod-ca-cert\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.567060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.561711 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-podinfo\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.567060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.561886 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-data\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.567060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.561909 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgqcp\" (UniqueName: \"kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-kube-api-access-lgqcp\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.567060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.562006 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-socket\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.567060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.562107 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-token\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.567060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.562134 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-certs\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.676959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.665024 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-token\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.677282 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.677252 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-certs\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.677926 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.677905 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-envoy\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.678148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.678131 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-credential-socket\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.678247 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.678234 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istiod-ca-cert\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.678350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.678337 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-podinfo\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.678583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.678565 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-data\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.678711 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.678696 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgqcp\" (UniqueName: \"kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-kube-api-access-lgqcp\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.678900 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.678886 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-socket\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.679476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.679454 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-socket\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.679619 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.677751 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-certs\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.685956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.680673 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-credential-socket\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.685956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.680966 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-data\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.685956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.681995 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istiod-ca-cert\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.685956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.683334 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-envoy\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.696352 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.696314 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-podinfo\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.713626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.713583 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-token\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.733158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.733122 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgqcp\" (UniqueName: \"kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-kube-api-access-lgqcp\") pod \"e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:25.825137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:25.824987 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:26.188224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:26.188192 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg"] Oct 30 18:19:26.189827 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:19:26.189795 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a0c1ed_a134_4e64_ad6f_1a5d3a7cdd83.slice/crio-f5a6d8c51b8c19f8b0b7a1360c5d8f9eeb29268ec330fbbe635b88267bd3a209 WatchSource:0}: Error finding container f5a6d8c51b8c19f8b0b7a1360c5d8f9eeb29268ec330fbbe635b88267bd3a209: Status 404 returned error can't find the container with id f5a6d8c51b8c19f8b0b7a1360c5d8f9eeb29268ec330fbbe635b88267bd3a209 Oct 30 18:19:26.192698 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:26.192655 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:26.192810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:26.192735 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:26.192810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:26.192764 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:26.861528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:26.861485 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" event={"ID":"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83","Type":"ContainerStarted","Data":"481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5"} Oct 30 18:19:26.861981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:26.861549 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" event={"ID":"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83","Type":"ContainerStarted","Data":"f5a6d8c51b8c19f8b0b7a1360c5d8f9eeb29268ec330fbbe635b88267bd3a209"} Oct 30 18:19:26.915505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:26.915437 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" podStartSLOduration=1.915421609 podStartE2EDuration="1.915421609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:26.913029182 +0000 UTC m=+1625.870033019" watchObservedRunningTime="2026-10-30 18:19:26.915421609 +0000 UTC m=+1625.872425445" Oct 30 18:19:27.825318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:27.825254 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:27.831348 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:27.831318 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:27.868179 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:27.868145 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:27.869192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:27.869173 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:19:29.601240 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.601208 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:29.642644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.642615 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:19:29.681990 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.681909 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-credential-socket\") pod \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " Oct 30 18:19:29.682161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.682062 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-certs\") pod \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " Oct 30 18:19:29.682256 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.682243 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-token\" (UniqueName: \"kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-token\") pod \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " Oct 30 18:19:29.682313 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.682255 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-credential-socket" pod "e44e1f4a-a663-4903-8a3c-0237dd2c3643" (UID: "e44e1f4a-a663-4903-8a3c-0237dd2c3643"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.682313 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.682286 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-socket\") pod \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " Oct 30 18:19:29.682425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.682315 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-certs" pod "e44e1f4a-a663-4903-8a3c-0237dd2c3643" (UID: "e44e1f4a-a663-4903-8a3c-0237dd2c3643"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.682425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.682333 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-kube-api-access-vwhgt\" (UniqueName: \"kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-kube-api-access-vwhgt\") pod \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " Oct 30 18:19:29.682527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.682425 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-envoy\") pod \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " Oct 30 18:19:29.682527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.682485 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-socket" pod "e44e1f4a-a663-4903-8a3c-0237dd2c3643" (UID: "e44e1f4a-a663-4903-8a3c-0237dd2c3643"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.682527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.682492 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-podinfo\") pod \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " Oct 30 18:19:29.682666 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.682542 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istiod-ca-cert\") pod \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " Oct 30 18:19:29.682666 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.682621 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-data\") pod \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\" (UID: \"e44e1f4a-a663-4903-8a3c-0237dd2c3643\") " Oct 30 18:19:29.683248 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.683215 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-data" pod "e44e1f4a-a663-4903-8a3c-0237dd2c3643" (UID: "e44e1f4a-a663-4903-8a3c-0237dd2c3643"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.683374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.683246 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.683374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.683252 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istiod-ca-cert" pod "e44e1f4a-a663-4903-8a3c-0237dd2c3643" (UID: "e44e1f4a-a663-4903-8a3c-0237dd2c3643"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:29.683374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.683272 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.683374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.683289 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.685375 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.685345 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-envoy" pod "e44e1f4a-a663-4903-8a3c-0237dd2c3643" (UID: "e44e1f4a-a663-4903-8a3c-0237dd2c3643"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.685586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.685552 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-podinfo" pod "e44e1f4a-a663-4903-8a3c-0237dd2c3643" (UID: "e44e1f4a-a663-4903-8a3c-0237dd2c3643"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:29.685698 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.685610 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-kube-api-access-vwhgt" pod "e44e1f4a-a663-4903-8a3c-0237dd2c3643" (UID: "e44e1f4a-a663-4903-8a3c-0237dd2c3643"). InnerVolumeSpecName "kube-api-access-vwhgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:29.685833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.685804 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-token" pod "e44e1f4a-a663-4903-8a3c-0237dd2c3643" (UID: "e44e1f4a-a663-4903-8a3c-0237dd2c3643"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:29.783952 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.783901 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07abf63a-7209-4001-90dc-af2cb30525ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07abf63a-7209-4001-90dc-af2cb30525ec-istio-podinfo\") pod \"07abf63a-7209-4001-90dc-af2cb30525ec\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " Oct 30 18:19:29.784137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.783981 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-certs\") pod \"07abf63a-7209-4001-90dc-af2cb30525ec\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " Oct 30 18:19:29.784137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.784083 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-istio-token\" (UniqueName: \"kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-istio-token\") pod \"07abf63a-7209-4001-90dc-af2cb30525ec\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " Oct 30 18:19:29.784137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.784134 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-kube-api-access-67b8v\" (UniqueName: \"kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-kube-api-access-67b8v\") pod \"07abf63a-7209-4001-90dc-af2cb30525ec\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " Oct 30 18:19:29.784324 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.784212 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-data\") pod \"07abf63a-7209-4001-90dc-af2cb30525ec\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " Oct 30 18:19:29.784324 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.784246 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-envoy\") pod \"07abf63a-7209-4001-90dc-af2cb30525ec\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " Oct 30 18:19:29.784324 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.784277 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-certs" pod "07abf63a-7209-4001-90dc-af2cb30525ec" (UID: "07abf63a-7209-4001-90dc-af2cb30525ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.784512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.784355 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07abf63a-7209-4001-90dc-af2cb30525ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07abf63a-7209-4001-90dc-af2cb30525ec-istiod-ca-cert\") pod \"07abf63a-7209-4001-90dc-af2cb30525ec\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " Oct 30 18:19:29.784512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.784397 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-socket\") pod \"07abf63a-7209-4001-90dc-af2cb30525ec\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " Oct 30 18:19:29.784512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.784482 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-credential-socket\") pod \"07abf63a-7209-4001-90dc-af2cb30525ec\" (UID: \"07abf63a-7209-4001-90dc-af2cb30525ec\") " Oct 30 18:19:29.784512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.784490 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-data" pod "07abf63a-7209-4001-90dc-af2cb30525ec" (UID: "07abf63a-7209-4001-90dc-af2cb30525ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.784791 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.784718 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-socket" pod "07abf63a-7209-4001-90dc-af2cb30525ec" (UID: "07abf63a-7209-4001-90dc-af2cb30525ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.784791 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.784771 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-credential-socket" pod "07abf63a-7209-4001-90dc-af2cb30525ec" (UID: "07abf63a-7209-4001-90dc-af2cb30525ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.785264 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.785214 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.785264 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.785240 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.785264 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.785255 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.785481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.785269 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.785481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.785282 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwhgt\" (UniqueName: \"kubernetes.io/projected/e44e1f4a-a663-4903-8a3c-0237dd2c3643-kube-api-access-vwhgt\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.785481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.785296 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.785481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.785308 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.785481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.785320 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.785481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.785333 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.785481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.785345 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44e1f4a-a663-4903-8a3c-0237dd2c3643-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.785973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.785610 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07abf63a-7209-4001-90dc-af2cb30525ec-istiod-ca-cert" pod "07abf63a-7209-4001-90dc-af2cb30525ec" (UID: "07abf63a-7209-4001-90dc-af2cb30525ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:29.786777 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.786745 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-istio-token" pod "07abf63a-7209-4001-90dc-af2cb30525ec" (UID: "07abf63a-7209-4001-90dc-af2cb30525ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:29.786993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.786967 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07abf63a-7209-4001-90dc-af2cb30525ec-istio-podinfo" pod "07abf63a-7209-4001-90dc-af2cb30525ec" (UID: "07abf63a-7209-4001-90dc-af2cb30525ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:29.787131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.787002 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-kube-api-access-67b8v" pod "07abf63a-7209-4001-90dc-af2cb30525ec" (UID: "07abf63a-7209-4001-90dc-af2cb30525ec"). InnerVolumeSpecName "kube-api-access-67b8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:29.787331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.787309 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-envoy" pod "07abf63a-7209-4001-90dc-af2cb30525ec" (UID: "07abf63a-7209-4001-90dc-af2cb30525ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:29.884766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.884728 4883 generic.go:356] "Generic (PLEG): container finished" podID="07abf63a-7209-4001-90dc-af2cb30525ec" containerID="2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a" exitCode=0 Oct 30 18:19:29.884995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.884779 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" Oct 30 18:19:29.884995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.884808 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" event={"ID":"07abf63a-7209-4001-90dc-af2cb30525ec","Type":"ContainerDied","Data":"2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a"} Oct 30 18:19:29.884995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.884865 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8" event={"ID":"07abf63a-7209-4001-90dc-af2cb30525ec","Type":"ContainerDied","Data":"317e73bc97ef67fd406e0e1dc1e6a6541a347051c24fbf5a55c4bdf78bad2b50"} Oct 30 18:19:29.884995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.884896 4883 scope.go:122] "RemoveContainer" containerID="2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a" Oct 30 18:19:29.886456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.886423 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07abf63a-7209-4001-90dc-af2cb30525ec-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.886456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.886453 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07abf63a-7209-4001-90dc-af2cb30525ec-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.886637 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.886462 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.886637 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.886476 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67b8v\" (UniqueName: \"kubernetes.io/projected/07abf63a-7209-4001-90dc-af2cb30525ec-kube-api-access-67b8v\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.886637 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.886486 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07abf63a-7209-4001-90dc-af2cb30525ec-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:19:29.886776 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.886754 4883 generic.go:356] "Generic (PLEG): container finished" podID="e44e1f4a-a663-4903-8a3c-0237dd2c3643" containerID="8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9" exitCode=0 Oct 30 18:19:29.886833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.886787 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" event={"ID":"e44e1f4a-a663-4903-8a3c-0237dd2c3643","Type":"ContainerDied","Data":"8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9"} Oct 30 18:19:29.886833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.886816 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" Oct 30 18:19:29.886909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.886844 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz" event={"ID":"e44e1f4a-a663-4903-8a3c-0237dd2c3643","Type":"ContainerDied","Data":"2f801893d1a619de471a3023415822e6512aa72b46531c71d5caf4ed360fa9bb"} Oct 30 18:19:29.897525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.897502 4883 scope.go:122] "RemoveContainer" containerID="2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a" Oct 30 18:19:29.897901 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:19:29.897867 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a\": container with ID starting with 2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a not found: ID does not exist" containerID="2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a" Oct 30 18:19:29.898093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.897909 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a"} err="failed to get container status \"2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a\": rpc error: code = NotFound desc = could not find container \"2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a\": container with ID starting with 2dc0b4dfb89e9ed9f673807d6c9ca31cb91059619d38ec67314f3c89e2a9b25a not found: ID does not exist" Oct 30 18:19:29.898093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.897950 4883 scope.go:122] "RemoveContainer" containerID="8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9" Oct 30 18:19:29.908431 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.908406 4883 scope.go:122] "RemoveContainer" containerID="8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9" Oct 30 18:19:29.908754 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:19:29.908732 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9\": container with ID starting with 8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9 not found: ID does not exist" containerID="8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9" Oct 30 18:19:29.908803 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.908764 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9"} err="failed to get container status \"8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9\": rpc error: code = NotFound desc = could not find container \"8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9\": container with ID starting with 8dc3d78609856e0fc977ea657edcd9ec01edca7871276aee8d5b09348af798b9 not found: ID does not exist" Oct 30 18:19:29.950742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.950708 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8"] Oct 30 18:19:29.953770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.953741 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-lxhx8"] Oct 30 18:19:29.990000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.989967 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz"] Oct 30 18:19:29.993225 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:29.993180 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c6b2120f-openshift-default-bd58474f7-w6mcz"] Oct 30 18:19:31.180146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:31.180107 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07abf63a-7209-4001-90dc-af2cb30525ec" path="/var/lib/kubelet/pods/07abf63a-7209-4001-90dc-af2cb30525ec/volumes" Oct 30 18:19:31.180961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:31.180919 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44e1f4a-a663-4903-8a3c-0237dd2c3643" path="/var/lib/kubelet/pods/e44e1f4a-a663-4903-8a3c-0237dd2c3643/volumes" Oct 30 18:19:35.081440 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:19:35.081399 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:19:58.876978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.876923 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6"] Oct 30 18:19:58.878296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.878263 4883 cpu_manager.go:414] "containerMap: removing container" podUID="07abf63a-7209-4001-90dc-af2cb30525ec" containerName="istio-proxy" Oct 30 18:19:58.878296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.878296 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07abf63a-7209-4001-90dc-af2cb30525ec" containerName="istio-proxy" Oct 30 18:19:58.878511 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.878338 4883 cpu_manager.go:414] "containerMap: removing container" podUID="e44e1f4a-a663-4903-8a3c-0237dd2c3643" containerName="istio-proxy" Oct 30 18:19:58.878511 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.878346 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44e1f4a-a663-4903-8a3c-0237dd2c3643" containerName="istio-proxy" Oct 30 18:19:58.878511 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.878499 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="07abf63a-7209-4001-90dc-af2cb30525ec" containerName="istio-proxy" Oct 30 18:19:58.878659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.878517 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44e1f4a-a663-4903-8a3c-0237dd2c3643" containerName="istio-proxy" Oct 30 18:19:58.882891 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.882862 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:58.904317 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.904281 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-394a31-openshift-default-dockercfg-46lmq\"" Oct 30 18:19:58.919927 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.919885 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6"] Oct 30 18:19:58.968996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.968951 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4141ed82-3540-42d3-9346-368784ca91c0-istiod-ca-cert\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:58.969177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.969005 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-socket\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:58.969225 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.969160 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-certs\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:58.969284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.969261 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv2sh\" (UniqueName: \"kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-kube-api-access-vv2sh\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:58.969320 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.969313 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-credential-socket\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:58.969366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.969352 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4141ed82-3540-42d3-9346-368784ca91c0-istio-podinfo\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:58.969517 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.969499 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-data\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:58.969642 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.969563 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-envoy\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:58.969642 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:58.969620 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-istio-token\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.073527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.073490 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-envoy\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.073747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.073559 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-istio-token\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.073747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.073682 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4141ed82-3540-42d3-9346-368784ca91c0-istiod-ca-cert\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.073747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.073720 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-socket\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.073915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.073817 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-certs\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.074030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.073908 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv2sh\" (UniqueName: \"kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-kube-api-access-vv2sh\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.079996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.074760 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4141ed82-3540-42d3-9346-368784ca91c0-istiod-ca-cert\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.079996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.074808 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-socket\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.079996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.074823 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-credential-socket\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.079996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.074901 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4141ed82-3540-42d3-9346-368784ca91c0-istio-podinfo\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.079996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.074991 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-data\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.079996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.075812 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-certs\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.079996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.076784 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-data\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.079996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.077313 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-credential-socket\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.080955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.080913 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-envoy\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.117242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.108747 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4141ed82-3540-42d3-9346-368784ca91c0-istio-podinfo\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.122657 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.119205 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv2sh\" (UniqueName: \"kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-kube-api-access-vv2sh\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.158492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.132542 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-istio-token\") pod \"e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.236068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.234206 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:19:59.630474 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.630445 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6"] Oct 30 18:19:59.632669 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:19:59.632632 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4141ed82_3540_42d3_9346_368784ca91c0.slice/crio-b970e1cde5b684257274423fb720f98ec42593b689eb3719999a852a82e15e59 WatchSource:0}: Error finding container b970e1cde5b684257274423fb720f98ec42593b689eb3719999a852a82e15e59: Status 404 returned error can't find the container with id b970e1cde5b684257274423fb720f98ec42593b689eb3719999a852a82e15e59 Oct 30 18:19:59.635415 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.635374 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:59.635502 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.635452 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:59.635502 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:19:59.635484 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:00.185501 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:00.185456 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" event={"ID":"4141ed82-3540-42d3-9346-368784ca91c0","Type":"ContainerStarted","Data":"1470e8d33a92d55ef75b0ef800f94d0951bd36aa76528c7f3e89e12eeed9901f"} Oct 30 18:20:00.186095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:00.185533 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" event={"ID":"4141ed82-3540-42d3-9346-368784ca91c0","Type":"ContainerStarted","Data":"b970e1cde5b684257274423fb720f98ec42593b689eb3719999a852a82e15e59"} Oct 30 18:20:00.234897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:00.234857 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:20:00.241384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:00.241349 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:20:00.270131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:00.270062 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" podStartSLOduration=2.270038924 podStartE2EDuration="2.270038924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:00.253363076 +0000 UTC m=+1659.210366913" watchObservedRunningTime="2026-10-30 18:20:00.270038924 +0000 UTC m=+1659.227042760" Oct 30 18:20:01.193616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:01.193581 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:20:01.194923 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:01.194898 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:20:05.090532 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:20:05.090413 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:20:18.793189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:18.793132 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:19.064267 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:19.064125 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:19.111665 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:19.111068 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:20.555966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:20.555584 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 18:20:20.959740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:20.959485 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:20:21.180962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:21.154831 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:20:21.296253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:21.296146 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:20:21.615369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:21.614574 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:21.626060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:21.621452 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:20:21.664010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:21.653003 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:20:21.664010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:21.659470 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:21.698747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:21.698701 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:21.747234 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:21.747186 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.123122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.123065 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.124076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.124048 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:20:22.168089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.168032 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:20:22.170414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.168358 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.199897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.198059 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:20:22.199897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.199003 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:20:22.351960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.340132 4883 scope.go:122] "RemoveContainer" containerID="964e7dc20e5ab39956a29f3515db9381dfab447f8812e0dd1e608dab6191e662" Oct 30 18:20:22.459961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.435046 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.544772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.482504 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:20:22.567991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.558972 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:20:22.590952 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.589914 4883 scope.go:122] "RemoveContainer" containerID="52064480904fc5871d78deabef5b8aecab38eb9981ca23c730b39d1d6fdf6add" Oct 30 18:20:22.643021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.635102 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:20:22.796276 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.650099 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.800489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.800449 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:20:22.839157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.664767 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:20:22.839648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.839624 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:20:22.841469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.749222 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:20:22.846142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.655488 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.846365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.701195 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.846534 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.841395 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:20:22.847865 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.847829 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:20:22.850542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.850517 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:20:22.873002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.850859 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:20:22.891212 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.891156 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:22.891690 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.891564 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:22.990979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:22.976177 4883 scope.go:122] "RemoveContainer" containerID="20bfdea67675d12f0b88763effa7da352aa343c47845919594b615726aaec628" Oct 30 18:20:23.143806 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.143760 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.198164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.198047 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.213977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.199914 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:20:23.216991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.215130 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:20:23.232722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.230711 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:20:23.561178 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.561129 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.562039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.562006 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.724394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.723224 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:20:23.724394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.723315 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:20:23.865720 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.865656 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:20:23.868105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.868048 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.897182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.888929 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.916959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.887916 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:20:23.916959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.910069 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:23.942117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:23.942061 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.384678 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.384613 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.385712 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.385671 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:20:24.416950 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.416651 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:20:24.452636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.452586 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:20:24.454468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.454439 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:20:24.472961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.460334 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.487670 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.487231 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.661016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.659166 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.661016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.659257 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:20:24.682184 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.674789 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:20:24.879314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.867917 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.959542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.959432 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:24.960175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.959609 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:20:24.960607 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:24.960292 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.389962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:25.389056 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:20:25.431961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:25.431024 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:20:25.493077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:25.493024 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.688023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:25.667046 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.688023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:25.667314 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:20:25.688023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:25.667713 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.710962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:25.693038 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.871909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:25.871865 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:20:25.894658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:25.894613 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.974019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:25.966346 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:20:25.974019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:25.966845 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:20:25.974019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:25.967213 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:20:26.421966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.420918 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:20:26.451988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.448689 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:26.466980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.464233 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:26.522964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.496955 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:20:26.650521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.650465 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 30 18:20:26.677145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.677034 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:26.679901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.678571 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:26.697079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.697030 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:26.698016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.697990 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:20:26.934021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.900427 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.170382 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.936017 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.174961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.937188 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded" Oct 30 18:20:27.174961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.983727 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:20:27.174961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.983781 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.174961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:26.983913 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:20:27.174961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.162502 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:20:27.273857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.273731 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:20:27.291970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.291150 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:20:27.630076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.557712 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:20:27.651007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.648065 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:20:27.703237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.702060 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.703237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.702664 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.721957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.719437 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:20:27.725227 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.724973 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.755652 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.755623 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:20:27.758696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.758671 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:20:27.760906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.760882 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:20:27.829058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.829011 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:20:27.833286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.833254 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:20:27.833998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.833973 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:20:27.834955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:27.834920 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:20:35.084793 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:20:35.084755 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:20:35.166525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:35.166485 4883 generic.go:356] "Generic (PLEG): container finished" podID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerID="481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5" exitCode=0 Oct 30 18:20:35.166730 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:35.166565 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" event={"ID":"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83","Type":"ContainerDied","Data":"481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5"} Oct 30 18:20:35.167221 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:35.167201 4883 scope.go:122] "RemoveContainer" containerID="481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5" Oct 30 18:20:35.169251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:35.169214 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:35.169347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:35.169307 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:35.169390 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:35.169351 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:36.229772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:36.229728 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" event={"ID":"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83","Type":"ContainerStarted","Data":"c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed"} Oct 30 18:20:36.826005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:36.825957 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:20:36.826205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:36.826179 4883 patch_prober.go:29] interesting pod/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:20:36.826273 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:36.826247 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 18:20:37.826425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:37.826382 4883 patch_prober.go:29] interesting pod/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:20:37.826850 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:37.826454 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 18:20:38.825432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:38.825392 4883 patch_prober.go:29] interesting pod/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:20:38.825612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:38.825448 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 18:20:39.217661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.217624 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz"] Oct 30 18:20:39.221347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.221320 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.241247 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.241217 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-189cda-openshift-default-dockercfg-kv2k4\"" Oct 30 18:20:39.261188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.261145 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz"] Oct 30 18:20:39.297488 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.297307 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-envoy\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.297488 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.297380 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-token\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.297488 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.297482 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hbmk\" (UniqueName: \"kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-kube-api-access-5hbmk\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.297824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.297523 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istiod-ca-cert\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.297824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.297561 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-credential-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.297824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.297622 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.297824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.297671 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-podinfo\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.297824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.297741 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-certs\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.298109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.297984 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-data\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.399819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.399774 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.400036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.399921 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-podinfo\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.400036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.399995 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-certs\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.400163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.400145 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-data\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.400221 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.400209 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-envoy\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.400276 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.400242 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-token\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.400329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.400272 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.400389 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.400349 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hbmk\" (UniqueName: \"kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-kube-api-access-5hbmk\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.400444 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.400394 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istiod-ca-cert\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.400444 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.400436 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-credential-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.400821 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.400782 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-data\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.400821 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.400804 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-credential-socket\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.401095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.401068 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-certs\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.401599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.401579 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istiod-ca-cert\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.402776 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.402756 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-envoy\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.402925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.402907 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-podinfo\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.426691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.426652 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-token\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.426889 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.426864 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hbmk\" (UniqueName: \"kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-kube-api-access-5hbmk\") pod \"e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.559701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.557763 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:39.860895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.860410 4883 patch_prober.go:29] interesting pod/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:20:39.860895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:39.860468 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 18:20:40.048771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:40.048723 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:40.048928 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:40.048822 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:40.048928 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:40.048875 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:40.050303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:40.050267 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz"] Oct 30 18:20:40.287615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:40.286638 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" event={"ID":"3f2ec0d6-e3d4-445f-bc69-054410b322d1","Type":"ContainerStarted","Data":"ca8ba65d35baa670e03d3af57a6b75a789cbd16bf6470027d08b9f7cf0cb105e"} Oct 30 18:20:40.287615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:40.286709 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" event={"ID":"3f2ec0d6-e3d4-445f-bc69-054410b322d1","Type":"ContainerStarted","Data":"6f06d9fcea48c02c08aa1294be82017ddc1d57cd41e392d0ba71c9873f4fc099"} Oct 30 18:20:40.339579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:40.339513 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" podStartSLOduration=1.3394916399999999 podStartE2EDuration="1.33949164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:40.33448598 +0000 UTC m=+1699.291489904" watchObservedRunningTime="2026-10-30 18:20:40.33949164 +0000 UTC m=+1699.296495477" Oct 30 18:20:40.558556 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:40.558464 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:40.825883 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:40.825800 4883 patch_prober.go:29] interesting pod/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:20:40.825883 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:40.825865 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 18:20:41.318826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.318782 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x"] Oct 30 18:20:41.346499 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.346461 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x"] Oct 30 18:20:41.346856 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.346794 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.375390 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.363821 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e0bc49-openshift-default-dockercfg-jcrtt\"" Oct 30 18:20:41.513967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.489045 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-credential-socket\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.513967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.489164 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-podinfo\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.513967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.489321 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-envoy\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.513967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.489354 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-data\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.513967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.489462 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istiod-ca-cert\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.513967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.489514 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-socket\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.513967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.510268 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-certs\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.513967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.510321 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-token\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.513967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.510464 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v2xv\" (UniqueName: \"kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-kube-api-access-5v2xv\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.671906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.656954 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v2xv\" (UniqueName: \"kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-kube-api-access-5v2xv\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.671906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.657460 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:20:41.671906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.657518 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" podUID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 18:20:41.699963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.697141 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-credential-socket\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.699963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.697210 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-podinfo\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.699963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.697325 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-envoy\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.699963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.697355 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-data\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.699963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.697433 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istiod-ca-cert\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.699963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.697490 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-socket\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.699963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.697592 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-certs\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.699963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.697617 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-token\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.699963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.698720 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-credential-socket\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.708959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.704822 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istiod-ca-cert\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.731956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.709517 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-socket\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.731956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.709991 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-certs\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.731956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.710252 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-data\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.767504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.767421 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-token\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.782958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.780340 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-podinfo\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.802777 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.802716 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-envoy\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.820975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.814592 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v2xv\" (UniqueName: \"kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-kube-api-access-5v2xv\") pod \"e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:41.856975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.851086 4883 patch_prober.go:29] interesting pod/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:20:41.856975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:41.851152 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 18:20:42.039016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:42.038769 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:42.576048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:42.575422 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:20:42.576048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:42.575483 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" podUID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 18:20:42.814964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:42.810753 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:42.831102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:42.826847 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:42.831102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:42.827060 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:42.834902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:42.834868 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x"] Oct 30 18:20:42.877218 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:42.861693 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:20:42.877218 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:42.863333 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:20:42.877218 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:42.874457 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:20:43.558690 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:43.558650 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:20:43.558865 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:43.558718 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" podUID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 18:20:43.659584 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:43.659543 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" event={"ID":"32faa9f8-f224-4587-b81b-7ea9988fe4c6","Type":"ContainerStarted","Data":"1f4bbe2ccfb0e2b38056c7750ee695ada94847b2aed15f44c1ee94dbafd7a49f"} Oct 30 18:20:43.660078 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:43.659608 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" event={"ID":"32faa9f8-f224-4587-b81b-7ea9988fe4c6","Type":"ContainerStarted","Data":"fc1a5e57ba4b2443d4f32efec7cae09c747d2707dea184420d33f0741a84ad5a"} Oct 30 18:20:43.719119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:43.719046 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" podStartSLOduration=2.719025199 podStartE2EDuration="2.719025199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:43.709897012 +0000 UTC m=+1702.666900849" watchObservedRunningTime="2026-10-30 18:20:43.719025199 +0000 UTC m=+1702.676029035" Oct 30 18:20:44.039544 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:44.039499 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:44.048729 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:44.048699 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:44.558534 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:44.558487 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:20:44.558721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:44.558554 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" podUID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 18:20:44.693898 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:44.693857 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:44.695249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:44.695218 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:20:45.558867 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:45.558821 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:20:45.559106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:45.558977 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" podUID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 18:20:46.558744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:46.558697 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 18:20:46.559271 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:46.558818 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" podUID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 18:20:47.563443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:47.563408 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:47.563918 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:47.563690 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:47.565198 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:47.565169 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:20:48.364842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:48.364806 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt"] Oct 30 18:20:48.367869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:48.367847 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" Oct 30 18:20:48.391407 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:48.391359 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e0bc49-dockercfg-5dsff\"" Oct 30 18:20:48.393788 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:48.393639 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e0bc49\"" Oct 30 18:20:48.394134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:48.394111 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt"] Oct 30 18:20:48.648445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:48.643598 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a90af2-5579-450a-9597-513a50296cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt\" (UID: \"e0a90af2-5579-450a-9597-513a50296cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" Oct 30 18:20:48.648445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:48.643836 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qbtn\" (UniqueName: \"kubernetes.io/projected/e0a90af2-5579-450a-9597-513a50296cfe-kube-api-access-4qbtn\") pod \"payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt\" (UID: \"e0a90af2-5579-450a-9597-513a50296cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" Oct 30 18:20:48.745719 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:48.745674 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a90af2-5579-450a-9597-513a50296cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt\" (UID: \"e0a90af2-5579-450a-9597-513a50296cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" Oct 30 18:20:48.745910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:48.745859 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qbtn\" (UniqueName: \"kubernetes.io/projected/e0a90af2-5579-450a-9597-513a50296cfe-kube-api-access-4qbtn\") pod \"payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt\" (UID: \"e0a90af2-5579-450a-9597-513a50296cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" Oct 30 18:20:48.746636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:48.746612 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a90af2-5579-450a-9597-513a50296cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt\" (UID: \"e0a90af2-5579-450a-9597-513a50296cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" Oct 30 18:20:48.771891 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:48.771464 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qbtn\" (UniqueName: \"kubernetes.io/projected/e0a90af2-5579-450a-9597-513a50296cfe-kube-api-access-4qbtn\") pod \"payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt\" (UID: \"e0a90af2-5579-450a-9597-513a50296cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" Oct 30 18:20:48.981633 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:48.981542 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" Oct 30 18:20:49.295980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.295948 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt"] Oct 30 18:20:49.297749 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:20:49.297715 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a90af2_5579_450a_9597_513a50296cfe.slice/crio-601c178881cf6953c7167798fee0360d45ec971a02d719d0b41fb537230d437f WatchSource:0}: Error finding container 601c178881cf6953c7167798fee0360d45ec971a02d719d0b41fb537230d437f: Status 404 returned error can't find the container with id 601c178881cf6953c7167798fee0360d45ec971a02d719d0b41fb537230d437f Oct 30 18:20:49.299889 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.299871 4883 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:20:49.323842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.323805 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt"] Oct 30 18:20:49.327437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.327417 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:49.343764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.343722 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:20:49.343764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.343722 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:20:49.344032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.343804 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:20:49.357397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.357358 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:20:49.363018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.362975 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt"] Oct 30 18:20:49.363192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.363159 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt\" (UID: \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:49.363282 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.363214 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-466cf\" (UniqueName: \"kubernetes.io/projected/d025d4e8-e034-4fa4-baed-2c66e97ee35a-kube-api-access-466cf\") pod \"maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt\" (UID: \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:49.363350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.363290 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt\" (UID: \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:49.464825 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.464784 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt\" (UID: \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:49.465090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.464847 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-466cf\" (UniqueName: \"kubernetes.io/projected/d025d4e8-e034-4fa4-baed-2c66e97ee35a-kube-api-access-466cf\") pod \"maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt\" (UID: \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:49.465090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.464894 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt\" (UID: \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:49.465646 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.465619 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt\" (UID: \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:49.467520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.467491 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt\" (UID: \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:49.489987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.489923 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-466cf\" (UniqueName: \"kubernetes.io/projected/d025d4e8-e034-4fa4-baed-2c66e97ee35a-kube-api-access-466cf\") pod \"maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt\" (UID: \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:49.644990 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.641678 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:49.824502 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.824458 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" event={"ID":"e0a90af2-5579-450a-9597-513a50296cfe","Type":"ContainerStarted","Data":"601c178881cf6953c7167798fee0360d45ec971a02d719d0b41fb537230d437f"} Oct 30 18:20:49.977504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:49.977469 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt"] Oct 30 18:20:49.978892 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:20:49.978858 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd025d4e8_e034_4fa4_baed_2c66e97ee35a.slice/crio-06544c34ff693e82c6414150f023bedc2ac2edd09ec9daf0fa6d5acbe2693f2f WatchSource:0}: Error finding container 06544c34ff693e82c6414150f023bedc2ac2edd09ec9daf0fa6d5acbe2693f2f: Status 404 returned error can't find the container with id 06544c34ff693e82c6414150f023bedc2ac2edd09ec9daf0fa6d5acbe2693f2f Oct 30 18:20:50.836016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:50.835972 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" event={"ID":"d025d4e8-e034-4fa4-baed-2c66e97ee35a","Type":"ContainerStarted","Data":"b7b4b43beb6e98b4b6140a7edc3ab152e06fb868f5a3e64b7817dfcf9d653e88"} Oct 30 18:20:50.836531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:50.836062 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" event={"ID":"d025d4e8-e034-4fa4-baed-2c66e97ee35a","Type":"ContainerStarted","Data":"06544c34ff693e82c6414150f023bedc2ac2edd09ec9daf0fa6d5acbe2693f2f"} Oct 30 18:20:50.885543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:50.885482 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" podStartSLOduration=1.885464454 podStartE2EDuration="1.885464454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:50.882636123 +0000 UTC m=+1709.839639964" watchObservedRunningTime="2026-10-30 18:20:50.885464454 +0000 UTC m=+1709.842468289" Oct 30 18:20:51.847115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:51.847072 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" event={"ID":"e0a90af2-5579-450a-9597-513a50296cfe","Type":"ContainerStarted","Data":"877031e92db6aa3ca332d00d52421f5c336fa4f57cbf8fa0a1c4661b25523bfb"} Oct 30 18:20:51.847616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:51.847370 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" Oct 30 18:20:51.850358 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:51.850323 4883 generic.go:356] "Generic (PLEG): container finished" podID="d025d4e8-e034-4fa4-baed-2c66e97ee35a" containerID="b7b4b43beb6e98b4b6140a7edc3ab152e06fb868f5a3e64b7817dfcf9d653e88" exitCode=0 Oct 30 18:20:51.850513 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:51.850492 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" event={"ID":"d025d4e8-e034-4fa4-baed-2c66e97ee35a","Type":"ContainerDied","Data":"b7b4b43beb6e98b4b6140a7edc3ab152e06fb868f5a3e64b7817dfcf9d653e88"} Oct 30 18:20:51.892396 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:51.892332 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" podStartSLOduration=1.6794553620000001 podStartE2EDuration="3.892312558s" totalImagesPullingTime="2.212857196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:51.889748613 +0000 UTC m=+1710.846752451" watchObservedRunningTime="2026-10-30 18:20:51.892312558 +0000 UTC m=+1710.849316402" Oct 30 18:20:53.027418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.027389 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:53.124904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.124781 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d025d4e8-e034-4fa4-baed-2c66e97ee35a-kube-api-access-466cf\" (UniqueName: \"kubernetes.io/projected/d025d4e8-e034-4fa4-baed-2c66e97ee35a-kube-api-access-466cf\") pod \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\" (UID: \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\") " Oct 30 18:20:53.124904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.124842 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-xks-ca\") pod \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\" (UID: \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\") " Oct 30 18:20:53.124904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.124889 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-openshift-ca\") pod \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\" (UID: \"d025d4e8-e034-4fa4-baed-2c66e97ee35a\") " Oct 30 18:20:53.125672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.125640 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-openshift-ca" pod "d025d4e8-e034-4fa4-baed-2c66e97ee35a" (UID: "d025d4e8-e034-4fa4-baed-2c66e97ee35a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:53.127710 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.127679 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-xks-ca" pod "d025d4e8-e034-4fa4-baed-2c66e97ee35a" (UID: "d025d4e8-e034-4fa4-baed-2c66e97ee35a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:53.127814 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.127725 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d025d4e8-e034-4fa4-baed-2c66e97ee35a-kube-api-access-466cf" pod "d025d4e8-e034-4fa4-baed-2c66e97ee35a" (UID: "d025d4e8-e034-4fa4-baed-2c66e97ee35a"). InnerVolumeSpecName "kube-api-access-466cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:53.226848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.226807 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-466cf\" (UniqueName: \"kubernetes.io/projected/d025d4e8-e034-4fa4-baed-2c66e97ee35a-kube-api-access-466cf\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:20:53.226848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.226837 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:20:53.226848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.226847 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d025d4e8-e034-4fa4-baed-2c66e97ee35a-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:20:53.868124 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.868090 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" Oct 30 18:20:53.868456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.868119 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt" event={"ID":"d025d4e8-e034-4fa4-baed-2c66e97ee35a","Type":"ContainerDied","Data":"06544c34ff693e82c6414150f023bedc2ac2edd09ec9daf0fa6d5acbe2693f2f"} Oct 30 18:20:53.868562 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:20:53.868539 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06544c34ff693e82c6414150f023bedc2ac2edd09ec9daf0fa6d5acbe2693f2f" Oct 30 18:21:02.860480 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:02.860445 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" Oct 30 18:21:03.750639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:03.750600 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt"] Oct 30 18:21:03.772103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:03.772065 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-064408-fb2xt"] Oct 30 18:21:05.081079 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:21:05.081042 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:21:05.181620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:05.181576 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d025d4e8-e034-4fa4-baed-2c66e97ee35a" path="/var/lib/kubelet/pods/d025d4e8-e034-4fa4-baed-2c66e97ee35a/volumes" Oct 30 18:21:07.750426 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.750384 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq"] Oct 30 18:21:07.751058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.751029 4883 cpu_manager.go:414] "containerMap: removing container" podUID="d025d4e8-e034-4fa4-baed-2c66e97ee35a" containerName="revoke-keys" Oct 30 18:21:07.751058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.751047 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d025d4e8-e034-4fa4-baed-2c66e97ee35a" containerName="revoke-keys" Oct 30 18:21:07.751173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.751164 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="d025d4e8-e034-4fa4-baed-2c66e97ee35a" containerName="revoke-keys" Oct 30 18:21:07.754573 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.754549 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:07.775727 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.775689 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq"] Oct 30 18:21:07.784479 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.784443 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:21:07.784681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.784644 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:21:07.784805 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.784757 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:21:07.785376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.785358 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:21:07.920026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.913232 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq\" (UID: \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:07.920415 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.920379 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhv82\" (UniqueName: \"kubernetes.io/projected/47f43ea9-304d-40cd-b3d6-e26072dcb51d-kube-api-access-mhv82\") pod \"maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq\" (UID: \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:07.920606 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:07.920590 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq\" (UID: \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:08.022026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:08.021892 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq\" (UID: \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:08.022203 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:08.022169 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhv82\" (UniqueName: \"kubernetes.io/projected/47f43ea9-304d-40cd-b3d6-e26072dcb51d-kube-api-access-mhv82\") pod \"maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq\" (UID: \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:08.022275 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:08.022254 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq\" (UID: \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:08.022784 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:08.022752 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq\" (UID: \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:08.024775 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:08.024746 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq\" (UID: \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:08.047019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:08.046983 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhv82\" (UniqueName: \"kubernetes.io/projected/47f43ea9-304d-40cd-b3d6-e26072dcb51d-kube-api-access-mhv82\") pod \"maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq\" (UID: \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:08.069764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:08.069730 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:08.402181 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:21:08.397623 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f43ea9_304d_40cd_b3d6_e26072dcb51d.slice/crio-4a9182dc2302a2148cf37e6660290d8cb413c08fc19e49b206736a885a4ce1dc WatchSource:0}: Error finding container 4a9182dc2302a2148cf37e6660290d8cb413c08fc19e49b206736a885a4ce1dc: Status 404 returned error can't find the container with id 4a9182dc2302a2148cf37e6660290d8cb413c08fc19e49b206736a885a4ce1dc Oct 30 18:21:08.403825 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:08.403776 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq"] Oct 30 18:21:08.987212 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:08.987108 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" event={"ID":"47f43ea9-304d-40cd-b3d6-e26072dcb51d","Type":"ContainerStarted","Data":"61d5abddae738cc7782261f60897e119ea3ebecc701b166dbc5e3ba55ed5d2b6"} Oct 30 18:21:08.987212 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:08.987177 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" event={"ID":"47f43ea9-304d-40cd-b3d6-e26072dcb51d","Type":"ContainerStarted","Data":"4a9182dc2302a2148cf37e6660290d8cb413c08fc19e49b206736a885a4ce1dc"} Oct 30 18:21:09.038078 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:09.038025 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" podStartSLOduration=2.03800934 podStartE2EDuration="2.03800934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:09.036432828 +0000 UTC m=+1727.993436665" watchObservedRunningTime="2026-10-30 18:21:09.03800934 +0000 UTC m=+1727.995013178" Oct 30 18:21:09.995283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:09.995245 4883 generic.go:356] "Generic (PLEG): container finished" podID="47f43ea9-304d-40cd-b3d6-e26072dcb51d" containerID="61d5abddae738cc7782261f60897e119ea3ebecc701b166dbc5e3ba55ed5d2b6" exitCode=0 Oct 30 18:21:09.995742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:09.995390 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" event={"ID":"47f43ea9-304d-40cd-b3d6-e26072dcb51d","Type":"ContainerDied","Data":"61d5abddae738cc7782261f60897e119ea3ebecc701b166dbc5e3ba55ed5d2b6"} Oct 30 18:21:11.339122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:11.339079 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:11.407046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:11.407009 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-openshift-ca\") pod \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\" (UID: \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\") " Oct 30 18:21:11.407254 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:11.407057 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f43ea9-304d-40cd-b3d6-e26072dcb51d-kube-api-access-mhv82\" (UniqueName: \"kubernetes.io/projected/47f43ea9-304d-40cd-b3d6-e26072dcb51d-kube-api-access-mhv82\") pod \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\" (UID: \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\") " Oct 30 18:21:11.407254 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:11.407085 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-xks-ca\") pod \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\" (UID: \"47f43ea9-304d-40cd-b3d6-e26072dcb51d\") " Oct 30 18:21:11.407474 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:11.407446 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-openshift-ca" pod "47f43ea9-304d-40cd-b3d6-e26072dcb51d" (UID: "47f43ea9-304d-40cd-b3d6-e26072dcb51d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:11.407713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:11.407688 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:21:11.409529 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:11.409493 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f43ea9-304d-40cd-b3d6-e26072dcb51d-kube-api-access-mhv82" pod "47f43ea9-304d-40cd-b3d6-e26072dcb51d" (UID: "47f43ea9-304d-40cd-b3d6-e26072dcb51d"). InnerVolumeSpecName "kube-api-access-mhv82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:11.411311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:11.411286 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-xks-ca" pod "47f43ea9-304d-40cd-b3d6-e26072dcb51d" (UID: "47f43ea9-304d-40cd-b3d6-e26072dcb51d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:11.508956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:11.508898 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhv82\" (UniqueName: \"kubernetes.io/projected/47f43ea9-304d-40cd-b3d6-e26072dcb51d-kube-api-access-mhv82\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:21:11.509140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:11.508977 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f43ea9-304d-40cd-b3d6-e26072dcb51d-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:21:12.011574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:12.011544 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" Oct 30 18:21:12.011792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:12.011581 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq" event={"ID":"47f43ea9-304d-40cd-b3d6-e26072dcb51d","Type":"ContainerDied","Data":"4a9182dc2302a2148cf37e6660290d8cb413c08fc19e49b206736a885a4ce1dc"} Oct 30 18:21:12.011792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:12.011650 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a9182dc2302a2148cf37e6660290d8cb413c08fc19e49b206736a885a4ce1dc" Oct 30 18:21:24.872492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:24.872452 4883 scope.go:122] "RemoveContainer" containerID="11289482ef4c3d863f26f03703f14c2eef1f1373b520a1dd9dbad5561453c8d6" Oct 30 18:21:25.839518 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:25.839459 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq"] Oct 30 18:21:25.849682 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:25.849628 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-853d3a-xjkgq"] Oct 30 18:21:27.179009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:27.178973 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f43ea9-304d-40cd-b3d6-e26072dcb51d" path="/var/lib/kubelet/pods/47f43ea9-304d-40cd-b3d6-e26072dcb51d/volumes" Oct 30 18:21:33.097371 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.097288 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n"] Oct 30 18:21:33.097879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.097869 4883 cpu_manager.go:414] "containerMap: removing container" podUID="47f43ea9-304d-40cd-b3d6-e26072dcb51d" containerName="revoke-keys" Oct 30 18:21:33.097921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.097881 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f43ea9-304d-40cd-b3d6-e26072dcb51d" containerName="revoke-keys" Oct 30 18:21:33.098039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.098027 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f43ea9-304d-40cd-b3d6-e26072dcb51d" containerName="revoke-keys" Oct 30 18:21:33.105270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.105244 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:33.122106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.122075 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n"] Oct 30 18:21:33.131357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.131323 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:21:33.131560 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.131448 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:21:33.180691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.180661 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:21:33.180872 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.180715 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:21:33.190629 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.190587 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n\" (UID: \"48b24a38-9320-4941-aa5a-24cfbaf13c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:33.190820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.190692 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n\" (UID: \"48b24a38-9320-4941-aa5a-24cfbaf13c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:33.190820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.190790 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgrpf\" (UniqueName: \"kubernetes.io/projected/48b24a38-9320-4941-aa5a-24cfbaf13c65-kube-api-access-sgrpf\") pod \"maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n\" (UID: \"48b24a38-9320-4941-aa5a-24cfbaf13c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:33.292596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.292555 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n\" (UID: \"48b24a38-9320-4941-aa5a-24cfbaf13c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:33.292803 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.292623 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n\" (UID: \"48b24a38-9320-4941-aa5a-24cfbaf13c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:33.292803 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.292698 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgrpf\" (UniqueName: \"kubernetes.io/projected/48b24a38-9320-4941-aa5a-24cfbaf13c65-kube-api-access-sgrpf\") pod \"maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n\" (UID: \"48b24a38-9320-4941-aa5a-24cfbaf13c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:33.293364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.293336 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n\" (UID: \"48b24a38-9320-4941-aa5a-24cfbaf13c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:33.295241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.295218 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n\" (UID: \"48b24a38-9320-4941-aa5a-24cfbaf13c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:33.317625 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.317585 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgrpf\" (UniqueName: \"kubernetes.io/projected/48b24a38-9320-4941-aa5a-24cfbaf13c65-kube-api-access-sgrpf\") pod \"maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n\" (UID: \"48b24a38-9320-4941-aa5a-24cfbaf13c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:33.420795 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.420706 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:33.730900 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.730867 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n"] Oct 30 18:21:33.732607 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:21:33.732577 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b24a38_9320_4941_aa5a_24cfbaf13c65.slice/crio-a4ba778111be056c5b57406c13eba3effb2400bf734415bb4ce5208e7f7ae732 WatchSource:0}: Error finding container a4ba778111be056c5b57406c13eba3effb2400bf734415bb4ce5208e7f7ae732: Status 404 returned error can't find the container with id a4ba778111be056c5b57406c13eba3effb2400bf734415bb4ce5208e7f7ae732 Oct 30 18:21:33.916106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.916066 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr"] Oct 30 18:21:33.923131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.922574 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:33.957920 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:33.957885 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr"] Oct 30 18:21:34.015082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.014983 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr\" (UID: \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:34.015082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.015051 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dkm\" (UniqueName: \"kubernetes.io/projected/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-kube-api-access-24dkm\") pod \"maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr\" (UID: \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:34.015342 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.015178 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr\" (UID: \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:34.116726 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.116679 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr\" (UID: \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:34.117287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.116734 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dkm\" (UniqueName: \"kubernetes.io/projected/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-kube-api-access-24dkm\") pod \"maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr\" (UID: \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:34.117287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.116815 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr\" (UID: \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:34.117705 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.117679 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr\" (UID: \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:34.119238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.119220 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr\" (UID: \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:34.144405 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.144374 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dkm\" (UniqueName: \"kubernetes.io/projected/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-kube-api-access-24dkm\") pod \"maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr\" (UID: \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:34.204486 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.204448 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" event={"ID":"48b24a38-9320-4941-aa5a-24cfbaf13c65","Type":"ContainerStarted","Data":"4b383ca609316ee7cb9eeb4e1385e14f0522af367f5e8d61b7283fa646a3dfdc"} Oct 30 18:21:34.204686 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.204507 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" event={"ID":"48b24a38-9320-4941-aa5a-24cfbaf13c65","Type":"ContainerStarted","Data":"a4ba778111be056c5b57406c13eba3effb2400bf734415bb4ce5208e7f7ae732"} Oct 30 18:21:34.248562 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.248509 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" podStartSLOduration=1.248495586 podStartE2EDuration="1.248495586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:34.246441091 +0000 UTC m=+1753.203444927" watchObservedRunningTime="2026-10-30 18:21:34.248495586 +0000 UTC m=+1753.205499422" Oct 30 18:21:34.266009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.265899 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:34.575359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:34.575329 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr"] Oct 30 18:21:34.577124 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:21:34.577094 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d90602a_4b9b_4ec2_9fbf_dccd9a7fb70c.slice/crio-40e4e6bfb4e14cdf582b9bb263f7103117e7b6562424839733aebe18d51e6575 WatchSource:0}: Error finding container 40e4e6bfb4e14cdf582b9bb263f7103117e7b6562424839733aebe18d51e6575: Status 404 returned error can't find the container with id 40e4e6bfb4e14cdf582b9bb263f7103117e7b6562424839733aebe18d51e6575 Oct 30 18:21:35.081432 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:21:35.081391 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:21:35.212993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:35.212873 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" event={"ID":"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c","Type":"ContainerStarted","Data":"d91d81f6af607304345bf34ccff0d7ad46973fcd60496dc9e808fb234a3503d2"} Oct 30 18:21:35.212993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:35.212959 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" event={"ID":"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c","Type":"ContainerStarted","Data":"40e4e6bfb4e14cdf582b9bb263f7103117e7b6562424839733aebe18d51e6575"} Oct 30 18:21:35.214691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:35.214666 4883 generic.go:356] "Generic (PLEG): container finished" podID="48b24a38-9320-4941-aa5a-24cfbaf13c65" containerID="4b383ca609316ee7cb9eeb4e1385e14f0522af367f5e8d61b7283fa646a3dfdc" exitCode=0 Oct 30 18:21:35.214793 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:35.214700 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" event={"ID":"48b24a38-9320-4941-aa5a-24cfbaf13c65","Type":"ContainerDied","Data":"4b383ca609316ee7cb9eeb4e1385e14f0522af367f5e8d61b7283fa646a3dfdc"} Oct 30 18:21:35.257875 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:35.257820 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" podStartSLOduration=2.257806326 podStartE2EDuration="2.257806326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:35.255349384 +0000 UTC m=+1754.212353212" watchObservedRunningTime="2026-10-30 18:21:35.257806326 +0000 UTC m=+1754.214810198" Oct 30 18:21:36.223693 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:36.223651 4883 generic.go:356] "Generic (PLEG): container finished" podID="9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c" containerID="d91d81f6af607304345bf34ccff0d7ad46973fcd60496dc9e808fb234a3503d2" exitCode=0 Oct 30 18:21:36.224210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:36.223736 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" event={"ID":"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c","Type":"ContainerDied","Data":"d91d81f6af607304345bf34ccff0d7ad46973fcd60496dc9e808fb234a3503d2"} Oct 30 18:21:36.431277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:36.422411 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:36.514094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:36.514003 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-openshift-ca\") pod \"48b24a38-9320-4941-aa5a-24cfbaf13c65\" (UID: \"48b24a38-9320-4941-aa5a-24cfbaf13c65\") " Oct 30 18:21:36.514259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:36.514095 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-xks-ca\") pod \"48b24a38-9320-4941-aa5a-24cfbaf13c65\" (UID: \"48b24a38-9320-4941-aa5a-24cfbaf13c65\") " Oct 30 18:21:36.514259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:36.514166 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b24a38-9320-4941-aa5a-24cfbaf13c65-kube-api-access-sgrpf\" (UniqueName: \"kubernetes.io/projected/48b24a38-9320-4941-aa5a-24cfbaf13c65-kube-api-access-sgrpf\") pod \"48b24a38-9320-4941-aa5a-24cfbaf13c65\" (UID: \"48b24a38-9320-4941-aa5a-24cfbaf13c65\") " Oct 30 18:21:36.514466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:36.514438 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-openshift-ca" pod "48b24a38-9320-4941-aa5a-24cfbaf13c65" (UID: "48b24a38-9320-4941-aa5a-24cfbaf13c65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:36.514776 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:36.514757 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:21:36.516569 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:36.516538 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b24a38-9320-4941-aa5a-24cfbaf13c65-kube-api-access-sgrpf" pod "48b24a38-9320-4941-aa5a-24cfbaf13c65" (UID: "48b24a38-9320-4941-aa5a-24cfbaf13c65"). InnerVolumeSpecName "kube-api-access-sgrpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:36.516696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:36.516636 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-xks-ca" pod "48b24a38-9320-4941-aa5a-24cfbaf13c65" (UID: "48b24a38-9320-4941-aa5a-24cfbaf13c65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:36.616369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:36.616325 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b24a38-9320-4941-aa5a-24cfbaf13c65-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:21:36.616369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:36.616358 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgrpf\" (UniqueName: \"kubernetes.io/projected/48b24a38-9320-4941-aa5a-24cfbaf13c65-kube-api-access-sgrpf\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:21:37.236384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.236323 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" Oct 30 18:21:37.236384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.236324 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n" event={"ID":"48b24a38-9320-4941-aa5a-24cfbaf13c65","Type":"ContainerDied","Data":"a4ba778111be056c5b57406c13eba3effb2400bf734415bb4ce5208e7f7ae732"} Oct 30 18:21:37.236902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.236430 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4ba778111be056c5b57406c13eba3effb2400bf734415bb4ce5208e7f7ae732" Oct 30 18:21:37.427453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.427420 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:37.430980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.430909 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-openshift-ca\") pod \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\" (UID: \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\") " Oct 30 18:21:37.431224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.431206 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-xks-ca\") pod \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\" (UID: \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\") " Oct 30 18:21:37.431294 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.431255 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-kube-api-access-24dkm\" (UniqueName: \"kubernetes.io/projected/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-kube-api-access-24dkm\") pod \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\" (UID: \"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c\") " Oct 30 18:21:37.431294 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.431282 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-openshift-ca" pod "9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c" (UID: "9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:37.431736 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.431715 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:21:37.433646 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.433606 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-kube-api-access-24dkm" pod "9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c" (UID: "9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c"). InnerVolumeSpecName "kube-api-access-24dkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:21:37.433767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.433717 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-xks-ca" pod "9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c" (UID: "9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:37.532738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.532647 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:21:37.532738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:37.532682 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24dkm\" (UniqueName: \"kubernetes.io/projected/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c-kube-api-access-24dkm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:21:38.245448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:38.245415 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" Oct 30 18:21:38.245877 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:38.245437 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr" event={"ID":"9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c","Type":"ContainerDied","Data":"40e4e6bfb4e14cdf582b9bb263f7103117e7b6562424839733aebe18d51e6575"} Oct 30 18:21:38.245877 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:38.245513 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40e4e6bfb4e14cdf582b9bb263f7103117e7b6562424839733aebe18d51e6575" Oct 30 18:21:47.704089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:47.704044 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr"] Oct 30 18:21:47.721320 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:47.721277 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c33ee4-drmjr"] Oct 30 18:21:49.179825 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:49.179792 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c" path="/var/lib/kubelet/pods/9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c/volumes" Oct 30 18:21:55.308754 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.308706 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg"] Oct 30 18:21:55.309335 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.309307 4883 cpu_manager.go:414] "containerMap: removing container" podUID="9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c" containerName="revoke-keys" Oct 30 18:21:55.309335 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.309321 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c" containerName="revoke-keys" Oct 30 18:21:55.309413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.309358 4883 cpu_manager.go:414] "containerMap: removing container" podUID="48b24a38-9320-4941-aa5a-24cfbaf13c65" containerName="revoke-keys" Oct 30 18:21:55.309413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.309364 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b24a38-9320-4941-aa5a-24cfbaf13c65" containerName="revoke-keys" Oct 30 18:21:55.309492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.309479 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b24a38-9320-4941-aa5a-24cfbaf13c65" containerName="revoke-keys" Oct 30 18:21:55.309527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.309497 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d90602a-4b9b-4ec2-9fbf-dccd9a7fb70c" containerName="revoke-keys" Oct 30 18:21:55.313336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.313311 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:21:55.333669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.333633 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg"] Oct 30 18:21:55.342331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.342296 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:21:55.342331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.342296 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:21:55.366193 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.366159 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-069812-6lftg\" (UID: \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:21:55.366486 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.366290 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tz7s\" (UniqueName: \"kubernetes.io/projected/41743778-3a4c-4b5f-9ea9-1e6fcda72787-kube-api-access-4tz7s\") pod \"maas-api-revoke-keys-e2e-ipp-b-069812-6lftg\" (UID: \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:21:55.366486 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.366399 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-069812-6lftg\" (UID: \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:21:55.399126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.399089 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:21:55.399342 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.399133 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:21:55.467657 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.467618 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-069812-6lftg\" (UID: \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:21:55.467852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.467824 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-069812-6lftg\" (UID: \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:21:55.467903 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.467875 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tz7s\" (UniqueName: \"kubernetes.io/projected/41743778-3a4c-4b5f-9ea9-1e6fcda72787-kube-api-access-4tz7s\") pod \"maas-api-revoke-keys-e2e-ipp-b-069812-6lftg\" (UID: \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:21:55.468457 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.468433 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-069812-6lftg\" (UID: \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:21:55.470396 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.470366 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-069812-6lftg\" (UID: \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:21:55.492620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.492580 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tz7s\" (UniqueName: \"kubernetes.io/projected/41743778-3a4c-4b5f-9ea9-1e6fcda72787-kube-api-access-4tz7s\") pod \"maas-api-revoke-keys-e2e-ipp-b-069812-6lftg\" (UID: \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:21:55.628265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.628228 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:21:55.942396 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:55.942353 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg"] Oct 30 18:21:55.943890 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:21:55.943847 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41743778_3a4c_4b5f_9ea9_1e6fcda72787.slice/crio-7212958a26278984c63ebc2506ed7e67b0e1a08592da787ba5589d501561b1c4 WatchSource:0}: Error finding container 7212958a26278984c63ebc2506ed7e67b0e1a08592da787ba5589d501561b1c4: Status 404 returned error can't find the container with id 7212958a26278984c63ebc2506ed7e67b0e1a08592da787ba5589d501561b1c4 Oct 30 18:21:56.414103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:56.414063 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" event={"ID":"41743778-3a4c-4b5f-9ea9-1e6fcda72787","Type":"ContainerStarted","Data":"9ecf6fc546cf10e08068a321910d72030e83a26db4f9ba0fdbb126fd73f0183c"} Oct 30 18:21:56.414518 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:56.414122 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" event={"ID":"41743778-3a4c-4b5f-9ea9-1e6fcda72787","Type":"ContainerStarted","Data":"7212958a26278984c63ebc2506ed7e67b0e1a08592da787ba5589d501561b1c4"} Oct 30 18:21:56.460450 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:56.460391 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" podStartSLOduration=1.460372748 podStartE2EDuration="1.460372748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:56.456782852 +0000 UTC m=+1775.413786688" watchObservedRunningTime="2026-10-30 18:21:56.460372748 +0000 UTC m=+1775.417376584" Oct 30 18:21:57.446109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:57.446069 4883 generic.go:356] "Generic (PLEG): container finished" podID="41743778-3a4c-4b5f-9ea9-1e6fcda72787" containerID="9ecf6fc546cf10e08068a321910d72030e83a26db4f9ba0fdbb126fd73f0183c" exitCode=0 Oct 30 18:21:57.446984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:57.446953 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" event={"ID":"41743778-3a4c-4b5f-9ea9-1e6fcda72787","Type":"ContainerDied","Data":"9ecf6fc546cf10e08068a321910d72030e83a26db4f9ba0fdbb126fd73f0183c"} Oct 30 18:21:58.640199 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:58.639527 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94"] Oct 30 18:21:58.669069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:58.656573 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:21:58.854813 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:58.854768 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94"] Oct 30 18:21:58.973961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:58.966061 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3be60-zcp94\" (UID: \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:21:58.973961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:58.968996 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3be60-zcp94\" (UID: \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:21:58.973961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:58.969179 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zjzh\" (UniqueName: \"kubernetes.io/projected/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-kube-api-access-9zjzh\") pod \"maas-api-revoke-keys-e2e-api-a-a3be60-zcp94\" (UID: \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:21:59.094244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.094202 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3be60-zcp94\" (UID: \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:21:59.094587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.094564 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3be60-zcp94\" (UID: \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:21:59.094800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.094783 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjzh\" (UniqueName: \"kubernetes.io/projected/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-kube-api-access-9zjzh\") pod \"maas-api-revoke-keys-e2e-api-a-a3be60-zcp94\" (UID: \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:21:59.394960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.384905 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3be60-zcp94\" (UID: \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:21:59.557842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.557751 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a3be60-zcp94\" (UID: \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:21:59.614894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.614844 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:21:59.670535 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.670450 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:21:59.675445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.675407 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjzh\" (UniqueName: \"kubernetes.io/projected/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-kube-api-access-9zjzh\") pod \"maas-api-revoke-keys-e2e-api-a-a3be60-zcp94\" (UID: \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:21:59.702201 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.702167 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:21:59.842025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.841204 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-openshift-ca" pod "41743778-3a4c-4b5f-9ea9-1e6fcda72787" (UID: "41743778-3a4c-4b5f-9ea9-1e6fcda72787"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:21:59.842025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.841331 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-openshift-ca\") pod \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\" (UID: \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\") " Oct 30 18:21:59.842025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.841415 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-xks-ca\") pod \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\" (UID: \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\") " Oct 30 18:21:59.842025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.841659 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41743778-3a4c-4b5f-9ea9-1e6fcda72787-kube-api-access-4tz7s\" (UniqueName: \"kubernetes.io/projected/41743778-3a4c-4b5f-9ea9-1e6fcda72787-kube-api-access-4tz7s\") pod \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\" (UID: \"41743778-3a4c-4b5f-9ea9-1e6fcda72787\") " Oct 30 18:21:59.865961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.842440 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:21:59.865961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.864256 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:21:59.865961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.864749 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:21:59.897991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.897047 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:21:59.898418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.898370 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-xks-ca" pod "41743778-3a4c-4b5f-9ea9-1e6fcda72787" (UID: "41743778-3a4c-4b5f-9ea9-1e6fcda72787"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:21:59.924020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.922343 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:21:59.993991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.947259 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41743778-3a4c-4b5f-9ea9-1e6fcda72787-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:21:59.993991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.979870 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:21:59.993991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:21:59.984464 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:00.020786 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.013175 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41743778-3a4c-4b5f-9ea9-1e6fcda72787-kube-api-access-4tz7s" pod "41743778-3a4c-4b5f-9ea9-1e6fcda72787" (UID: "41743778-3a4c-4b5f-9ea9-1e6fcda72787"). InnerVolumeSpecName "kube-api-access-4tz7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:00.098990 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.072493 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tz7s\" (UniqueName: \"kubernetes.io/projected/41743778-3a4c-4b5f-9ea9-1e6fcda72787-kube-api-access-4tz7s\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:00.216024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.197283 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:00.498827 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.490083 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:22:00.498827 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.490539 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:22:00.532255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.531969 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:00.544158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.543842 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 18:22:00.582108 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.580808 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:00.627519 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.619470 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:00.627519 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.620117 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:22:00.696459 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.669678 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:00.884961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.871612 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:22:01.436997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:00.954560 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:01.437556 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:01.437462 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:22:01.619304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:01.352390 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:01.619304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:01.595108 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:01.677966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:01.661779 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:22:01.677966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:01.662210 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:01.815008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:01.798764 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:02.126152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.126101 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.126419 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.105550 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:02.126645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.013679 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.126836 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.111735 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:22:02.263765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.263651 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.435209 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.435147 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.435954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.435897 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:22:02.485570 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.485523 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.544032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.496602 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.627970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.614543 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.834101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.663717 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.834101 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:22:02.668484 4883 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.353s" Oct 30 18:22:02.834101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.708796 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:02.890051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.890016 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:22:02.890318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.890303 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:22:02.890463 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.890452 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:22:02.890573 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.890562 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:22:02.890704 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.890691 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:22:02.890831 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.890819 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:22:02.890984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.890958 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:22:02.891130 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.891117 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:22:02.891258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:02.891246 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:22:03.113375 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.112178 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.131094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.131041 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:22:03.132676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.131470 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:03.158976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.147244 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.274828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.274797 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" Oct 30 18:22:03.275959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.275085 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.453017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.438685 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.454429 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.454389 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.544917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.544861 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.651061 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.547806 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.724527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.724410 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94"] Oct 30 18:22:03.724833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.724789 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg" event={"ID":"41743778-3a4c-4b5f-9ea9-1e6fcda72787","Type":"ContainerDied","Data":"7212958a26278984c63ebc2506ed7e67b0e1a08592da787ba5589d501561b1c4"} Oct 30 18:22:03.725006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.724988 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7212958a26278984c63ebc2506ed7e67b0e1a08592da787ba5589d501561b1c4" Oct 30 18:22:03.725194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.725167 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:22:03.789975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.789909 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.817251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.817187 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:03.954027 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:03.925696 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.113899 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.113853 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.114860 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.114835 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:22:04.147578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.147529 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:22:04.163968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.147554 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.168960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.168013 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:04.306978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.285052 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:04.306978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.285124 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.306978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.285719 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:22:04.498333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.427019 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.498333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.456075 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:22:04.498333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.444027 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.675977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.662319 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:22:04.718393 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.704570 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" event={"ID":"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b","Type":"ContainerStarted","Data":"b42e451ae53520eb2a7c21589f336260c070d9f74c4a3ac972647414412bece2"} Oct 30 18:22:04.763871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.723092 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.792835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.792778 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.850371 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.849393 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:04.965417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:04.965368 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.163105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.163046 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.209964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.184983 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.234060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.233926 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.273963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.253480 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:22:05.301044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.300997 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.590060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.571323 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.625971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.601239 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.773971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.607068 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:22:05.775771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.665673 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.776772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.733580 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.824992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.824928 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.918806 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.918082 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:05.987962 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:22:05.969442 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:22:05.987962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:05.969600 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.108856 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.108816 4883 generic.go:356] "Generic (PLEG): container finished" podID="6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b" containerID="47cb41506d1b2c202811080aa922963f6fbc681126874707534d941ec863e0b9" exitCode=0 Oct 30 18:22:06.118272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.109526 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" event={"ID":"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b","Type":"ContainerDied","Data":"47cb41506d1b2c202811080aa922963f6fbc681126874707534d941ec863e0b9"} Oct 30 18:22:06.195023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.190419 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.222102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.222047 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.237094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.237036 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:06.264109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.264048 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.304114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.304055 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.575223 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.575080 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.651279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.651222 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 30 18:22:06.778748 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.776073 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.778748 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.777153 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:22:06.778748 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.777479 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.780705 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.780668 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp"] Oct 30 18:22:06.781891 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.781864 4883 cpu_manager.go:414] "containerMap: removing container" podUID="41743778-3a4c-4b5f-9ea9-1e6fcda72787" containerName="revoke-keys" Oct 30 18:22:06.782060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.782043 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41743778-3a4c-4b5f-9ea9-1e6fcda72787" containerName="revoke-keys" Oct 30 18:22:06.782291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.782277 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="41743778-3a4c-4b5f-9ea9-1e6fcda72787" containerName="revoke-keys" Oct 30 18:22:06.905177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.904023 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.938480 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.938433 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.954966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.954253 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp"] Oct 30 18:22:06.956525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.956161 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:06.956525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.956228 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" podUID="0086e9f0-1201-4263-b710-5417225349ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:06.979174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.979129 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:22:07.006001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:06.980985 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:07.006001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.002050 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:07.064366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.048596 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wws7s\" (UniqueName: \"kubernetes.io/projected/431e9454-d692-4cfa-9c64-4d8c656a556b-kube-api-access-wws7s\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp\" (UID: \"431e9454-d692-4cfa-9c64-4d8c656a556b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:07.064366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.048681 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp\" (UID: \"431e9454-d692-4cfa-9c64-4d8c656a556b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:07.064366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.048919 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp\" (UID: \"431e9454-d692-4cfa-9c64-4d8c656a556b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:07.238242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.238138 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:07.238520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.238480 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:22:07.238760 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.238726 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:07.311174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.309759 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:22:07.311174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.309836 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:22:07.311174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.310080 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp\" (UID: \"431e9454-d692-4cfa-9c64-4d8c656a556b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:07.311174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.310192 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wws7s\" (UniqueName: \"kubernetes.io/projected/431e9454-d692-4cfa-9c64-4d8c656a556b-kube-api-access-wws7s\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp\" (UID: \"431e9454-d692-4cfa-9c64-4d8c656a556b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:07.311174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.310262 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp\" (UID: \"431e9454-d692-4cfa-9c64-4d8c656a556b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:07.364007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.363973 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp\" (UID: \"431e9454-d692-4cfa-9c64-4d8c656a556b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:07.426553 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.387598 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp\" (UID: \"431e9454-d692-4cfa-9c64-4d8c656a556b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:07.676255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.667673 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:22:07.696784 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.696658 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wws7s\" (UniqueName: \"kubernetes.io/projected/431e9454-d692-4cfa-9c64-4d8c656a556b-kube-api-access-wws7s\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp\" (UID: \"431e9454-d692-4cfa-9c64-4d8c656a556b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:07.748182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.748139 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:07.791236 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.777056 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:07.805986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.803325 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:07.805986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.803773 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:22:07.915243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.915204 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:07.915459 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.915433 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:07.926105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.926063 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:07.926417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.926359 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:07.961972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:07.958083 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.433026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.423115 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:22:08.433026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.423507 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:22:08.433026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.423807 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:08.433026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.424095 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:22:08.527961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.523155 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Oct 30 18:22:08.614978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.612044 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.628647 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.628096 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:22:08.628647 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.628404 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:22:08.633613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.633570 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded" Oct 30 18:22:08.657554 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.657486 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:08.657867 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.657833 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.679998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.678979 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:22:08.816449 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.802068 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.816449 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.804085 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:08.816449 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.804360 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:22:09.010495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:08.977510 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:22:09.381623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.379625 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.381623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.380382 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Oct 30 18:22:09.771483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.429099 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.771483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.430801 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.771483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.439037 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.771483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.439362 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:22:09.771483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.453094 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.771483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.487509 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.771483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.622219 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.771483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.655254 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.771483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.655518 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.771483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.670109 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.771483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.691353 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:09.833955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.822585 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:22:09.833955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.822975 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:22:09.833955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.823248 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:22:09.958954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:09.954162 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded" Oct 30 18:22:10.046863 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.044735 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:10.168514 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.137019 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:10.168514 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.153179 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:22:10.197746 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:22:10.197708 4883 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.024s" Oct 30 18:22:10.212020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.197751 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" event={"ID":"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b","Type":"ContainerDied","Data":"b42e451ae53520eb2a7c21589f336260c070d9f74c4a3ac972647414412bece2"} Oct 30 18:22:10.212020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.197796 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b42e451ae53520eb2a7c21589f336260c070d9f74c4a3ac972647414412bece2" Oct 30 18:22:10.212020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.197812 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9"] Oct 30 18:22:10.212020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.197837 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c048b0b-644j9"] Oct 30 18:22:10.212020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.201163 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp"] Oct 30 18:22:10.212020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.209471 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n"] Oct 30 18:22:10.216918 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.216865 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-189cda-qfr9n"] Oct 30 18:22:10.346458 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.287381 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:22:10.346458 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.290284 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:22:10.346458 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.345378 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-xks-ca\") pod \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\" (UID: \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\") " Oct 30 18:22:10.346458 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.345559 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-openshift-ca\") pod \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\" (UID: \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\") " Oct 30 18:22:10.346458 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.345681 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-kube-api-access-9zjzh\" (UniqueName: \"kubernetes.io/projected/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-kube-api-access-9zjzh\") pod \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\" (UID: \"6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b\") " Oct 30 18:22:10.399370 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.399329 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-openshift-ca" pod "6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b" (UID: "6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:10.500015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.485053 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:10.500015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.485479 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:22:10.500015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.486785 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:22:10.546543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.538110 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:10.546543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.541235 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-xks-ca" pod "6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b" (UID: "6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:10.546543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.542116 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:22:10.644632 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.641005 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:22:10.659573 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.659403 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-kube-api-access-9zjzh" pod "6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b" (UID: "6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b"). InnerVolumeSpecName "kube-api-access-9zjzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:10.660996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.660562 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:10.755036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.754996 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:22:10.795311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.795265 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:11.052017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.031155 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:22:11.052017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:10.847523 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:22:11.052349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.052170 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:22:11.055780 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.055746 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:22:11.129058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.064737 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:11.129058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.064777 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:11.129058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.110703 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zjzh\" (UniqueName: \"kubernetes.io/projected/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b-kube-api-access-9zjzh\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:11.358694 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.358661 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94" Oct 30 18:22:11.359775 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.359744 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b24a38-9320-4941-aa5a-24cfbaf13c65" path="/var/lib/kubelet/pods/48b24a38-9320-4941-aa5a-24cfbaf13c65/volumes" Oct 30 18:22:11.360224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.360208 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77227337-d235-4baf-8b11-4bf73c735420" path="/var/lib/kubelet/pods/77227337-d235-4baf-8b11-4bf73c735420/volumes" Oct 30 18:22:11.360857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.360842 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:22:11.360949 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.360875 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" event={"ID":"431e9454-d692-4cfa-9c64-4d8c656a556b","Type":"ContainerStarted","Data":"38bf3f80c3503bbb909b2466aa04cda77101427de67c4350527091b9de1f87b4"} Oct 30 18:22:11.360949 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.360947 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:22:11.361073 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.360977 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:22:11.361073 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.361061 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:22:11.361250 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.361235 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:22:11.361306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:11.361276 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:22:12.386120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:12.385794 4883 generic.go:356] "Generic (PLEG): container finished" podID="431e9454-d692-4cfa-9c64-4d8c656a556b" containerID="956415fdd3ad2c301c9043100b1d66f3d9f00635d64dc3de2d17994aa49dd930" exitCode=0 Oct 30 18:22:12.386120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:12.386041 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" event={"ID":"431e9454-d692-4cfa-9c64-4d8c656a556b","Type":"ContainerDied","Data":"956415fdd3ad2c301c9043100b1d66f3d9f00635d64dc3de2d17994aa49dd930"} Oct 30 18:22:13.632521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:13.632488 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:13.697038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:13.696350 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431e9454-d692-4cfa-9c64-4d8c656a556b-kube-api-access-wws7s\" (UniqueName: \"kubernetes.io/projected/431e9454-d692-4cfa-9c64-4d8c656a556b-kube-api-access-wws7s\") pod \"431e9454-d692-4cfa-9c64-4d8c656a556b\" (UID: \"431e9454-d692-4cfa-9c64-4d8c656a556b\") " Oct 30 18:22:13.697038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:13.696514 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-xks-ca\") pod \"431e9454-d692-4cfa-9c64-4d8c656a556b\" (UID: \"431e9454-d692-4cfa-9c64-4d8c656a556b\") " Oct 30 18:22:13.697038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:13.696725 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-openshift-ca\") pod \"431e9454-d692-4cfa-9c64-4d8c656a556b\" (UID: \"431e9454-d692-4cfa-9c64-4d8c656a556b\") " Oct 30 18:22:13.699103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:13.699065 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-openshift-ca" pod "431e9454-d692-4cfa-9c64-4d8c656a556b" (UID: "431e9454-d692-4cfa-9c64-4d8c656a556b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:13.700563 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:13.700513 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431e9454-d692-4cfa-9c64-4d8c656a556b-kube-api-access-wws7s" pod "431e9454-d692-4cfa-9c64-4d8c656a556b" (UID: "431e9454-d692-4cfa-9c64-4d8c656a556b"). InnerVolumeSpecName "kube-api-access-wws7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:13.705252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:13.705188 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-xks-ca" pod "431e9454-d692-4cfa-9c64-4d8c656a556b" (UID: "431e9454-d692-4cfa-9c64-4d8c656a556b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:13.800013 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:13.799978 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:13.800013 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:13.800008 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wws7s\" (UniqueName: \"kubernetes.io/projected/431e9454-d692-4cfa-9c64-4d8c656a556b-kube-api-access-wws7s\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:13.800274 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:13.800025 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431e9454-d692-4cfa-9c64-4d8c656a556b-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:14.423953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:14.422834 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" event={"ID":"431e9454-d692-4cfa-9c64-4d8c656a556b","Type":"ContainerDied","Data":"38bf3f80c3503bbb909b2466aa04cda77101427de67c4350527091b9de1f87b4"} Oct 30 18:22:14.423953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:14.422969 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38bf3f80c3503bbb909b2466aa04cda77101427de67c4350527091b9de1f87b4" Oct 30 18:22:14.423953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:14.423077 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp" Oct 30 18:22:15.736137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:15.736091 4883 patch_prober.go:29] interesting pod/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:15.736689 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:15.736164 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:16.203996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:16.203908 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz"] Oct 30 18:22:16.216228 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:16.216181 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" podUID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" containerName="istio-proxy" containerID="cri-o://ca8ba65d35baa670e03d3af57a6b75a789cbd16bf6470027d08b9f7cf0cb105e" gracePeriod=30 Oct 30 18:22:17.236167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:17.236113 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:17.236587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:17.236214 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:17.616490 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:17.566411 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:22:17.616490 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:17.566476 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" podUID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:22:18.482218 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:18.482160 4883 generic.go:356] "Generic (PLEG): container finished" podID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerID="1f4bbe2ccfb0e2b38056c7750ee695ada94847b2aed15f44c1ee94dbafd7a49f" exitCode=0 Oct 30 18:22:18.482987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:18.482854 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" event={"ID":"32faa9f8-f224-4587-b81b-7ea9988fe4c6","Type":"ContainerDied","Data":"1f4bbe2ccfb0e2b38056c7750ee695ada94847b2aed15f44c1ee94dbafd7a49f"} Oct 30 18:22:18.483650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:18.483628 4883 scope.go:122] "RemoveContainer" containerID="1f4bbe2ccfb0e2b38056c7750ee695ada94847b2aed15f44c1ee94dbafd7a49f" Oct 30 18:22:18.488651 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:18.488286 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:22:18.489323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:18.489281 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:22:18.489528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:18.489508 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:22:19.504161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:19.504124 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" event={"ID":"32faa9f8-f224-4587-b81b-7ea9988fe4c6","Type":"ContainerStarted","Data":"687cc4202164770f159edab241dcd7f4bfa8922fe437583daa3d17fbe31d59b4"} Oct 30 18:22:19.506159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:19.506131 4883 generic.go:356] "Generic (PLEG): container finished" podID="4141ed82-3540-42d3-9346-368784ca91c0" containerID="1470e8d33a92d55ef75b0ef800f94d0951bd36aa76528c7f3e89e12eeed9901f" exitCode=0 Oct 30 18:22:19.506318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:19.506206 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" event={"ID":"4141ed82-3540-42d3-9346-368784ca91c0","Type":"ContainerDied","Data":"1470e8d33a92d55ef75b0ef800f94d0951bd36aa76528c7f3e89e12eeed9901f"} Oct 30 18:22:19.506666 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:19.506649 4883 scope.go:122] "RemoveContainer" containerID="1470e8d33a92d55ef75b0ef800f94d0951bd36aa76528c7f3e89e12eeed9901f" Oct 30 18:22:19.508378 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:19.508344 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:22:19.508501 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:19.508437 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:22:19.508501 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:19.508488 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:22:20.052323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:20.051175 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:22:20.052323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:20.051304 4883 patch_prober.go:29] interesting pod/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 18:22:20.052323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:20.051347 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 18:22:20.522007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:20.521962 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" event={"ID":"4141ed82-3540-42d3-9346-368784ca91c0","Type":"ContainerStarted","Data":"f9801000bdc104610306a9af1cead8766dac7d1736f30d196413925fe2cf9238"} Oct 30 18:22:21.040016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.039979 4883 patch_prober.go:29] interesting pod/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 18:22:21.040217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.040035 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 18:22:21.235288 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.235237 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:22:21.235478 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.235256 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:22:21.235478 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.235378 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 18:22:21.636181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.636107 4883 generic.go:356] "Generic (PLEG): container finished" podID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" containerID="ca8ba65d35baa670e03d3af57a6b75a789cbd16bf6470027d08b9f7cf0cb105e" exitCode=0 Oct 30 18:22:21.636816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.636635 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" event={"ID":"3f2ec0d6-e3d4-445f-bc69-054410b322d1","Type":"ContainerDied","Data":"ca8ba65d35baa670e03d3af57a6b75a789cbd16bf6470027d08b9f7cf0cb105e"} Oct 30 18:22:21.657912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.656621 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:22:21.730113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.730079 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:22:21.742328 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.742296 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:22:21.743163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.743133 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:22:21.750713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.750679 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:22:21.804742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.799496 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-envoy\") pod \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " Oct 30 18:22:21.804742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.799564 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-credential-socket\") pod \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " Oct 30 18:22:21.804742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.799661 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istiod-ca-cert\") pod \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " Oct 30 18:22:21.804742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.799712 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-data\") pod \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " Oct 30 18:22:21.804742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.799765 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-certs\") pod \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " Oct 30 18:22:21.804742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.799880 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-podinfo\") pod \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " Oct 30 18:22:21.804742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.799965 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-kube-api-access-5hbmk\" (UniqueName: \"kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-kube-api-access-5hbmk\") pod \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " Oct 30 18:22:21.804742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.800025 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-socket\") pod \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " Oct 30 18:22:21.804742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.800080 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-token\" (UniqueName: \"kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-token\") pod \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\" (UID: \"3f2ec0d6-e3d4-445f-bc69-054410b322d1\") " Oct 30 18:22:21.805415 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.805368 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-data" pod "3f2ec0d6-e3d4-445f-bc69-054410b322d1" (UID: "3f2ec0d6-e3d4-445f-bc69-054410b322d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.827972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.826035 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-credential-socket" pod "3f2ec0d6-e3d4-445f-bc69-054410b322d1" (UID: "3f2ec0d6-e3d4-445f-bc69-054410b322d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.829079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.828464 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istiod-ca-cert" pod "3f2ec0d6-e3d4-445f-bc69-054410b322d1" (UID: "3f2ec0d6-e3d4-445f-bc69-054410b322d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:21.829079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.829033 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-certs" pod "3f2ec0d6-e3d4-445f-bc69-054410b322d1" (UID: "3f2ec0d6-e3d4-445f-bc69-054410b322d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.837013 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.835847 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-envoy" pod "3f2ec0d6-e3d4-445f-bc69-054410b322d1" (UID: "3f2ec0d6-e3d4-445f-bc69-054410b322d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.843034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.842972 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-token" pod "3f2ec0d6-e3d4-445f-bc69-054410b322d1" (UID: "3f2ec0d6-e3d4-445f-bc69-054410b322d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:21.847088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.847029 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-socket" pod "3f2ec0d6-e3d4-445f-bc69-054410b322d1" (UID: "3f2ec0d6-e3d4-445f-bc69-054410b322d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:21.858002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.855556 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-podinfo" pod "3f2ec0d6-e3d4-445f-bc69-054410b322d1" (UID: "3f2ec0d6-e3d4-445f-bc69-054410b322d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:21.858002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.857666 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-kube-api-access-5hbmk" pod "3f2ec0d6-e3d4-445f-bc69-054410b322d1" (UID: "3f2ec0d6-e3d4-445f-bc69-054410b322d1"). InnerVolumeSpecName "kube-api-access-5hbmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:21.913017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.907591 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.913017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.907635 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.913017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.907650 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.913017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.907665 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.913017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.907681 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.913017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.907693 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.913017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.907707 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f2ec0d6-e3d4-445f-bc69-054410b322d1-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.913017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.907722 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hbmk\" (UniqueName: \"kubernetes.io/projected/3f2ec0d6-e3d4-445f-bc69-054410b322d1-kube-api-access-5hbmk\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:21.913017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:21.907739 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f2ec0d6-e3d4-445f-bc69-054410b322d1-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:22.041743 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:22.041223 4883 patch_prober.go:29] interesting pod/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 18:22:22.041743 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:22.041308 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 18:22:22.235980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:22.235877 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:22:22.235980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:22.235952 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 18:22:22.646086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:22.646052 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" Oct 30 18:22:22.646086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:22.646061 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz" event={"ID":"3f2ec0d6-e3d4-445f-bc69-054410b322d1","Type":"ContainerDied","Data":"6f06d9fcea48c02c08aa1294be82017ddc1d57cd41e392d0ba71c9873f4fc099"} Oct 30 18:22:22.646641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:22.646157 4883 scope.go:122] "RemoveContainer" containerID="ca8ba65d35baa670e03d3af57a6b75a789cbd16bf6470027d08b9f7cf0cb105e" Oct 30 18:22:22.720467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:22.720410 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz"] Oct 30 18:22:22.724721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:22.724657 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-189cda-openshift-default-76776699c7-q9wvz"] Oct 30 18:22:23.039718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.039607 4883 patch_prober.go:29] interesting pod/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 18:22:23.040023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.039715 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 18:22:23.222989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.222167 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" path="/var/lib/kubelet/pods/3f2ec0d6-e3d4-445f-bc69-054410b322d1/volumes" Oct 30 18:22:23.235638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.235590 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:22:23.235913 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.235878 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 18:22:23.322024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.321008 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h"] Oct 30 18:22:23.322024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.321905 4883 cpu_manager.go:414] "containerMap: removing container" podUID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" containerName="istio-proxy" Oct 30 18:22:23.328962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.327957 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" containerName="istio-proxy" Oct 30 18:22:23.328962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.328080 4883 cpu_manager.go:414] "containerMap: removing container" podUID="6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b" containerName="revoke-keys" Oct 30 18:22:23.328962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.328093 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b" containerName="revoke-keys" Oct 30 18:22:23.328962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.328114 4883 cpu_manager.go:414] "containerMap: removing container" podUID="431e9454-d692-4cfa-9c64-4d8c656a556b" containerName="revoke-keys" Oct 30 18:22:23.328962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.328121 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431e9454-d692-4cfa-9c64-4d8c656a556b" containerName="revoke-keys" Oct 30 18:22:23.328962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.328318 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2ec0d6-e3d4-445f-bc69-054410b322d1" containerName="istio-proxy" Oct 30 18:22:23.328962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.328343 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b" containerName="revoke-keys" Oct 30 18:22:23.328962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.328355 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="431e9454-d692-4cfa-9c64-4d8c656a556b" containerName="revoke-keys" Oct 30 18:22:23.332138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.332107 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:23.336193 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.335834 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h"] Oct 30 18:22:23.349760 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.349732 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:22:23.350053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.349754 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:22:23.350170 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.349775 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:22:23.350268 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.349814 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:22:23.446189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.446157 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h\" (UID: \"0a29a83a-b938-4275-910c-8b71aa1e2295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:23.446522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.446509 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h\" (UID: \"0a29a83a-b938-4275-910c-8b71aa1e2295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:23.446635 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.446625 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmkkd\" (UniqueName: \"kubernetes.io/projected/0a29a83a-b938-4275-910c-8b71aa1e2295-kube-api-access-vmkkd\") pod \"maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h\" (UID: \"0a29a83a-b938-4275-910c-8b71aa1e2295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:23.548477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.548247 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h\" (UID: \"0a29a83a-b938-4275-910c-8b71aa1e2295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:23.548672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.548493 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h\" (UID: \"0a29a83a-b938-4275-910c-8b71aa1e2295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:23.548672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.548522 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmkkd\" (UniqueName: \"kubernetes.io/projected/0a29a83a-b938-4275-910c-8b71aa1e2295-kube-api-access-vmkkd\") pod \"maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h\" (UID: \"0a29a83a-b938-4275-910c-8b71aa1e2295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:23.549736 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.549708 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h\" (UID: \"0a29a83a-b938-4275-910c-8b71aa1e2295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:23.551904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.551375 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h\" (UID: \"0a29a83a-b938-4275-910c-8b71aa1e2295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:23.597967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.596099 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmkkd\" (UniqueName: \"kubernetes.io/projected/0a29a83a-b938-4275-910c-8b71aa1e2295-kube-api-access-vmkkd\") pod \"maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h\" (UID: \"0a29a83a-b938-4275-910c-8b71aa1e2295\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:23.652992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:23.652483 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:24.039788 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:24.039711 4883 patch_prober.go:29] interesting pod/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 18:22:24.039788 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:24.039758 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 18:22:24.141140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:24.135799 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h"] Oct 30 18:22:24.141140 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:22:24.140701 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a29a83a_b938_4275_910c_8b71aa1e2295.slice/crio-09648bfefdf72886f704521527b5223af5289f380865a04b62e579123c76c06e WatchSource:0}: Error finding container 09648bfefdf72886f704521527b5223af5289f380865a04b62e579123c76c06e: Status 404 returned error can't find the container with id 09648bfefdf72886f704521527b5223af5289f380865a04b62e579123c76c06e Oct 30 18:22:24.238443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:24.238408 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:22:24.238733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:24.238704 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 18:22:24.704075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:24.695670 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" event={"ID":"0a29a83a-b938-4275-910c-8b71aa1e2295","Type":"ContainerStarted","Data":"09648bfefdf72886f704521527b5223af5289f380865a04b62e579123c76c06e"} Oct 30 18:22:25.039929 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:25.039886 4883 patch_prober.go:29] interesting pod/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 18:22:25.040134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:25.039990 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 18:22:25.236107 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:25.236004 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:22:25.236107 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:25.236080 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 18:22:25.712070 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:25.712016 4883 generic.go:356] "Generic (PLEG): container finished" podID="0a29a83a-b938-4275-910c-8b71aa1e2295" containerID="6ed0d21db53aa001007609d78cd9e9f6d6f4a1185cec948ca84663a7ee16fd87" exitCode=0 Oct 30 18:22:25.712525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:25.712327 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" event={"ID":"0a29a83a-b938-4275-910c-8b71aa1e2295","Type":"ContainerDied","Data":"6ed0d21db53aa001007609d78cd9e9f6d6f4a1185cec948ca84663a7ee16fd87"} Oct 30 18:22:26.044087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:26.044043 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:22:26.044578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:26.044542 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:22:26.045493 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:26.045452 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:22:26.235096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:26.235060 4883 patch_prober.go:29] interesting pod/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 18:22:26.235531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:26.235493 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 18:22:26.570010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:26.569917 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp"] Oct 30 18:22:26.573363 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:26.573322 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8dc70-dz8bp"] Oct 30 18:22:27.110153 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.110124 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:27.198058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.196959 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="431e9454-d692-4cfa-9c64-4d8c656a556b" path="/var/lib/kubelet/pods/431e9454-d692-4cfa-9c64-4d8c656a556b/volumes" Oct 30 18:22:27.201552 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.201518 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg"] Oct 30 18:22:27.201787 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.201574 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-069812-6lftg"] Oct 30 18:22:27.228072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.228025 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-openshift-ca\") pod \"0a29a83a-b938-4275-910c-8b71aa1e2295\" (UID: \"0a29a83a-b938-4275-910c-8b71aa1e2295\") " Oct 30 18:22:27.228304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.228126 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-xks-ca\") pod \"0a29a83a-b938-4275-910c-8b71aa1e2295\" (UID: \"0a29a83a-b938-4275-910c-8b71aa1e2295\") " Oct 30 18:22:27.228304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.228261 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a29a83a-b938-4275-910c-8b71aa1e2295-kube-api-access-vmkkd\" (UniqueName: \"kubernetes.io/projected/0a29a83a-b938-4275-910c-8b71aa1e2295-kube-api-access-vmkkd\") pod \"0a29a83a-b938-4275-910c-8b71aa1e2295\" (UID: \"0a29a83a-b938-4275-910c-8b71aa1e2295\") " Oct 30 18:22:27.229671 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.229453 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-openshift-ca" pod "0a29a83a-b938-4275-910c-8b71aa1e2295" (UID: "0a29a83a-b938-4275-910c-8b71aa1e2295"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:27.231724 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.231693 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:27.241401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.234375 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a29a83a-b938-4275-910c-8b71aa1e2295-kube-api-access-vmkkd" pod "0a29a83a-b938-4275-910c-8b71aa1e2295" (UID: "0a29a83a-b938-4275-910c-8b71aa1e2295"). InnerVolumeSpecName "kube-api-access-vmkkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:27.249445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.249405 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:22:27.250652 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.250622 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:22:27.253178 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.253145 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:22:27.257473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.257424 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-xks-ca" pod "0a29a83a-b938-4275-910c-8b71aa1e2295" (UID: "0a29a83a-b938-4275-910c-8b71aa1e2295"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:27.333622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.333590 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmkkd\" (UniqueName: \"kubernetes.io/projected/0a29a83a-b938-4275-910c-8b71aa1e2295-kube-api-access-vmkkd\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:27.333861 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.333839 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a29a83a-b938-4275-910c-8b71aa1e2295-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:27.800299 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.800266 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" Oct 30 18:22:27.800299 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.800287 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h" event={"ID":"0a29a83a-b938-4275-910c-8b71aa1e2295","Type":"ContainerDied","Data":"09648bfefdf72886f704521527b5223af5289f380865a04b62e579123c76c06e"} Oct 30 18:22:27.800587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:27.800347 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09648bfefdf72886f704521527b5223af5289f380865a04b62e579123c76c06e" Oct 30 18:22:29.181145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:29.181113 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41743778-3a4c-4b5f-9ea9-1e6fcda72787" path="/var/lib/kubelet/pods/41743778-3a4c-4b5f-9ea9-1e6fcda72787/volumes" Oct 30 18:22:35.082055 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:22:35.082014 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:22:40.031966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:40.031914 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv"] Oct 30 18:22:40.054749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:40.054709 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2769dcb0-26ptv"] Oct 30 18:22:41.193736 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:41.193692 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d42db29-a73b-47ec-aa49-afadad615cfa" path="/var/lib/kubelet/pods/5d42db29-a73b-47ec-aa49-afadad615cfa/volumes" Oct 30 18:22:44.017246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:44.017186 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94"] Oct 30 18:22:44.021617 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:44.021585 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a3be60-zcp94"] Oct 30 18:22:45.182835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:45.182797 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b" path="/var/lib/kubelet/pods/6b4fbc7e-ab6f-4e62-a084-f9a8a23b958b/volumes" Oct 30 18:22:51.703983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:51.703927 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg"] Oct 30 18:22:51.704703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:51.704650 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" containerID="cri-o://c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed" gracePeriod=30 Oct 30 18:22:52.299004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.298929 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5"] Oct 30 18:22:52.299699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.299673 4883 cpu_manager.go:414] "containerMap: removing container" podUID="0a29a83a-b938-4275-910c-8b71aa1e2295" containerName="revoke-keys" Oct 30 18:22:52.299699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.299693 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a29a83a-b938-4275-910c-8b71aa1e2295" containerName="revoke-keys" Oct 30 18:22:52.299987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.299788 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a29a83a-b938-4275-910c-8b71aa1e2295" containerName="revoke-keys" Oct 30 18:22:52.303106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.303071 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:52.320351 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.320318 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:22:52.320523 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.320407 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:22:52.339097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.339043 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5"] Oct 30 18:22:52.349232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.349192 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4b2r\" (UniqueName: \"kubernetes.io/projected/53b1c58a-b359-4f57-a141-eeb36c130d4e-kube-api-access-c4b2r\") pod \"maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5\" (UID: \"53b1c58a-b359-4f57-a141-eeb36c130d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:52.349420 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.349333 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5\" (UID: \"53b1c58a-b359-4f57-a141-eeb36c130d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:52.349420 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.349357 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5\" (UID: \"53b1c58a-b359-4f57-a141-eeb36c130d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:52.388922 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.388891 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:22:52.388922 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.388892 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:22:52.450824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.450793 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5\" (UID: \"53b1c58a-b359-4f57-a141-eeb36c130d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:52.450824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.450827 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5\" (UID: \"53b1c58a-b359-4f57-a141-eeb36c130d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:52.451104 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.450998 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4b2r\" (UniqueName: \"kubernetes.io/projected/53b1c58a-b359-4f57-a141-eeb36c130d4e-kube-api-access-c4b2r\") pod \"maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5\" (UID: \"53b1c58a-b359-4f57-a141-eeb36c130d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:52.451599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.451572 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5\" (UID: \"53b1c58a-b359-4f57-a141-eeb36c130d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:52.453292 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.453269 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5\" (UID: \"53b1c58a-b359-4f57-a141-eeb36c130d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:52.476043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.476009 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4b2r\" (UniqueName: \"kubernetes.io/projected/53b1c58a-b359-4f57-a141-eeb36c130d4e-kube-api-access-c4b2r\") pod \"maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5\" (UID: \"53b1c58a-b359-4f57-a141-eeb36c130d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:52.624052 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.624012 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:52.934218 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:52.934179 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5"] Oct 30 18:22:52.934651 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:22:52.934565 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b1c58a_b359_4f57_a141_eeb36c130d4e.slice/crio-a548913579a1b7a0620b438efa28b8c9eb578b2b235adf243526afe9ebf89dd7 WatchSource:0}: Error finding container a548913579a1b7a0620b438efa28b8c9eb578b2b235adf243526afe9ebf89dd7: Status 404 returned error can't find the container with id a548913579a1b7a0620b438efa28b8c9eb578b2b235adf243526afe9ebf89dd7 Oct 30 18:22:53.059369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:53.059329 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" event={"ID":"53b1c58a-b359-4f57-a141-eeb36c130d4e","Type":"ContainerStarted","Data":"a548913579a1b7a0620b438efa28b8c9eb578b2b235adf243526afe9ebf89dd7"} Oct 30 18:22:54.068105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:54.068061 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" event={"ID":"53b1c58a-b359-4f57-a141-eeb36c130d4e","Type":"ContainerStarted","Data":"626f18d66ed1dd8775a84631877c105b003142ae3065bc30d842bbe3a3fc52a2"} Oct 30 18:22:54.112360 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:54.111258 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h"] Oct 30 18:22:54.114975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:54.114890 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" podStartSLOduration=2.1148670960000002 podStartE2EDuration="2.114867096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:54.111497001 +0000 UTC m=+1833.068500839" watchObservedRunningTime="2026-10-30 18:22:54.114867096 +0000 UTC m=+1833.071870934" Oct 30 18:22:54.123633 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:54.123570 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d951c4-jnm6h"] Oct 30 18:22:55.077742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:55.077696 4883 generic.go:356] "Generic (PLEG): container finished" podID="53b1c58a-b359-4f57-a141-eeb36c130d4e" containerID="626f18d66ed1dd8775a84631877c105b003142ae3065bc30d842bbe3a3fc52a2" exitCode=0 Oct 30 18:22:55.078235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:55.077786 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" event={"ID":"53b1c58a-b359-4f57-a141-eeb36c130d4e","Type":"ContainerDied","Data":"626f18d66ed1dd8775a84631877c105b003142ae3065bc30d842bbe3a3fc52a2"} Oct 30 18:22:55.181120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:55.181083 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a29a83a-b938-4275-910c-8b71aa1e2295" path="/var/lib/kubelet/pods/0a29a83a-b938-4275-910c-8b71aa1e2295/volumes" Oct 30 18:22:56.304060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:56.304031 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:56.418654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:56.418611 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b1c58a-b359-4f57-a141-eeb36c130d4e-kube-api-access-c4b2r\" (UniqueName: \"kubernetes.io/projected/53b1c58a-b359-4f57-a141-eeb36c130d4e-kube-api-access-c4b2r\") pod \"53b1c58a-b359-4f57-a141-eeb36c130d4e\" (UID: \"53b1c58a-b359-4f57-a141-eeb36c130d4e\") " Oct 30 18:22:56.418654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:56.418658 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-openshift-ca\") pod \"53b1c58a-b359-4f57-a141-eeb36c130d4e\" (UID: \"53b1c58a-b359-4f57-a141-eeb36c130d4e\") " Oct 30 18:22:56.418977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:56.418912 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-xks-ca\") pod \"53b1c58a-b359-4f57-a141-eeb36c130d4e\" (UID: \"53b1c58a-b359-4f57-a141-eeb36c130d4e\") " Oct 30 18:22:56.419123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:56.419100 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-openshift-ca" pod "53b1c58a-b359-4f57-a141-eeb36c130d4e" (UID: "53b1c58a-b359-4f57-a141-eeb36c130d4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:56.419563 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:56.419541 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:56.421048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:56.421023 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b1c58a-b359-4f57-a141-eeb36c130d4e-kube-api-access-c4b2r" pod "53b1c58a-b359-4f57-a141-eeb36c130d4e" (UID: "53b1c58a-b359-4f57-a141-eeb36c130d4e"). InnerVolumeSpecName "kube-api-access-c4b2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:56.421134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:56.421046 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-xks-ca" pod "53b1c58a-b359-4f57-a141-eeb36c130d4e" (UID: "53b1c58a-b359-4f57-a141-eeb36c130d4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:56.521344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:56.521310 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4b2r\" (UniqueName: \"kubernetes.io/projected/53b1c58a-b359-4f57-a141-eeb36c130d4e-kube-api-access-c4b2r\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:56.521344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:56.521341 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b1c58a-b359-4f57-a141-eeb36c130d4e-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:56.889968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:56.889924 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:22:57.034325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034222 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-data\") pod \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " Oct 30 18:22:57.034325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034318 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-certs\") pod \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " Oct 30 18:22:57.034684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034346 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istiod-ca-cert\") pod \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " Oct 30 18:22:57.034684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034366 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-credential-socket\") pod \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " Oct 30 18:22:57.034684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034391 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-kube-api-access-lgqcp\" (UniqueName: \"kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-kube-api-access-lgqcp\") pod \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " Oct 30 18:22:57.034684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034521 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-token\" (UniqueName: \"kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-token\") pod \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " Oct 30 18:22:57.034684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034550 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-socket\") pod \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " Oct 30 18:22:57.034684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034597 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-podinfo\") pod \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " Oct 30 18:22:57.034684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034585 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-data" pod "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" (UID: "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:57.034684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034646 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-certs" pod "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" (UID: "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:57.034684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034678 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-envoy\") pod \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\" (UID: \"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83\") " Oct 30 18:22:57.035230 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034724 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-credential-socket" pod "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" (UID: "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:57.035230 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034841 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-socket" pod "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" (UID: "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:57.035230 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.034889 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istiod-ca-cert" pod "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" (UID: "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:57.035512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.035490 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:57.035512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.035512 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:57.035640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.035521 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:57.035640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.035529 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:57.035640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.035539 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:57.036830 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.036805 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-envoy" pod "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" (UID: "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:57.037012 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.036950 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-podinfo" pod "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" (UID: "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:57.037303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.037273 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-kube-api-access-lgqcp" pod "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" (UID: "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83"). InnerVolumeSpecName "kube-api-access-lgqcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:57.037303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.037286 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-token" pod "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" (UID: "92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:57.093163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.093130 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" Oct 30 18:22:57.093362 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.093163 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5" event={"ID":"53b1c58a-b359-4f57-a141-eeb36c130d4e","Type":"ContainerDied","Data":"a548913579a1b7a0620b438efa28b8c9eb578b2b235adf243526afe9ebf89dd7"} Oct 30 18:22:57.093362 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.093223 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a548913579a1b7a0620b438efa28b8c9eb578b2b235adf243526afe9ebf89dd7" Oct 30 18:22:57.095293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.095265 4883 generic.go:356] "Generic (PLEG): container finished" podID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerID="c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed" exitCode=0 Oct 30 18:22:57.095469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.095316 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" Oct 30 18:22:57.095469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.095345 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" event={"ID":"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83","Type":"ContainerDied","Data":"c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed"} Oct 30 18:22:57.095469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.095402 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg" event={"ID":"92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83","Type":"ContainerDied","Data":"f5a6d8c51b8c19f8b0b7a1360c5d8f9eeb29268ec330fbbe635b88267bd3a209"} Oct 30 18:22:57.095469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.095432 4883 scope.go:122] "RemoveContainer" containerID="c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed" Oct 30 18:22:57.112432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.112411 4883 scope.go:122] "RemoveContainer" containerID="481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5" Oct 30 18:22:57.123052 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.123007 4883 scope.go:122] "RemoveContainer" containerID="c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed" Oct 30 18:22:57.123474 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:22:57.123452 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed\": container with ID starting with c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed not found: ID does not exist" containerID="c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed" Oct 30 18:22:57.123567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.123484 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed"} err="failed to get container status \"c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed\": rpc error: code = NotFound desc = could not find container \"c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed\": container with ID starting with c38ba054b3518ef042a88c4aca8d8d3590d5c2323a63417d8685b0b96e6a99ed not found: ID does not exist" Oct 30 18:22:57.123567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.123513 4883 scope.go:122] "RemoveContainer" containerID="481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5" Oct 30 18:22:57.123809 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:22:57.123792 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5\": container with ID starting with 481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5 not found: ID does not exist" containerID="481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5" Oct 30 18:22:57.123866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.123819 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5"} err="failed to get container status \"481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5\": rpc error: code = NotFound desc = could not find container \"481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5\": container with ID starting with 481c498993dc12a5a009af91b6bf8c1670f66cb0cbe8a0116619def7a3970ae5 not found: ID does not exist" Oct 30 18:22:57.137102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.137071 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:57.137102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.137099 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:57.137102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.137110 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgqcp\" (UniqueName: \"kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-kube-api-access-lgqcp\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:57.137338 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.137119 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:22:57.155767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.155678 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg"] Oct 30 18:22:57.174989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.168194 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a3be60-openshift-default-5778b886dc-d6mmg"] Oct 30 18:22:57.181104 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:22:57.181066 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" path="/var/lib/kubelet/pods/92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83/volumes" Oct 30 18:23:02.068796 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.068754 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5"] Oct 30 18:23:02.069367 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.069352 4883 cpu_manager.go:414] "containerMap: removing container" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" Oct 30 18:23:02.069412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.069368 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" Oct 30 18:23:02.069412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.069394 4883 cpu_manager.go:414] "containerMap: removing container" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" Oct 30 18:23:02.069412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.069399 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" Oct 30 18:23:02.069504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.069417 4883 cpu_manager.go:414] "containerMap: removing container" podUID="53b1c58a-b359-4f57-a141-eeb36c130d4e" containerName="revoke-keys" Oct 30 18:23:02.069504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.069424 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b1c58a-b359-4f57-a141-eeb36c130d4e" containerName="revoke-keys" Oct 30 18:23:02.069571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.069506 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" Oct 30 18:23:02.069571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.069519 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b1c58a-b359-4f57-a141-eeb36c130d4e" containerName="revoke-keys" Oct 30 18:23:02.074720 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.074697 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:02.094041 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.094008 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5"] Oct 30 18:23:02.112644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.112614 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:23:02.112836 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.112701 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:23:02.151723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.151688 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:23:02.152055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.152037 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:23:02.161445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.161402 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k48sz\" (UniqueName: \"kubernetes.io/projected/d286b388-ca68-4bda-b76d-07d6f362b101-kube-api-access-k48sz\") pod \"maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5\" (UID: \"d286b388-ca68-4bda-b76d-07d6f362b101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:02.161620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.161551 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5\" (UID: \"d286b388-ca68-4bda-b76d-07d6f362b101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:02.161699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.161683 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5\" (UID: \"d286b388-ca68-4bda-b76d-07d6f362b101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:02.263233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.263189 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5\" (UID: \"d286b388-ca68-4bda-b76d-07d6f362b101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:02.263596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.263575 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48sz\" (UniqueName: \"kubernetes.io/projected/d286b388-ca68-4bda-b76d-07d6f362b101-kube-api-access-k48sz\") pod \"maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5\" (UID: \"d286b388-ca68-4bda-b76d-07d6f362b101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:02.263721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.263684 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5\" (UID: \"d286b388-ca68-4bda-b76d-07d6f362b101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:02.264011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.263929 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5\" (UID: \"d286b388-ca68-4bda-b76d-07d6f362b101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:02.266047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.266021 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5\" (UID: \"d286b388-ca68-4bda-b76d-07d6f362b101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:02.290265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.290221 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k48sz\" (UniqueName: \"kubernetes.io/projected/d286b388-ca68-4bda-b76d-07d6f362b101-kube-api-access-k48sz\") pod \"maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5\" (UID: \"d286b388-ca68-4bda-b76d-07d6f362b101\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:02.389323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.389275 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:02.778003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.777909 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5"] Oct 30 18:23:02.783999 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:23:02.783919 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd286b388_ca68_4bda_b76d_07d6f362b101.slice/crio-3c6900ab86803a83aafdbe863d8ba04c6926b25cf0b5ec0e8cd6136a2aa97523 WatchSource:0}: Error finding container 3c6900ab86803a83aafdbe863d8ba04c6926b25cf0b5ec0e8cd6136a2aa97523: Status 404 returned error can't find the container with id 3c6900ab86803a83aafdbe863d8ba04c6926b25cf0b5ec0e8cd6136a2aa97523 Oct 30 18:23:02.952277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:02.952241 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt"] Oct 30 18:23:03.147789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:03.147757 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" event={"ID":"d286b388-ca68-4bda-b76d-07d6f362b101","Type":"ContainerStarted","Data":"3c6900ab86803a83aafdbe863d8ba04c6926b25cf0b5ec0e8cd6136a2aa97523"} Oct 30 18:23:04.157526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:04.157430 4883 generic.go:356] "Generic (PLEG): container finished" podID="d286b388-ca68-4bda-b76d-07d6f362b101" containerID="6f050e576fefdd5b6274e2e176a982096019c6c1bdd7d6c2ab5def663387bba8" exitCode=0 Oct 30 18:23:04.157526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:04.157497 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" event={"ID":"d286b388-ca68-4bda-b76d-07d6f362b101","Type":"ContainerDied","Data":"6f050e576fefdd5b6274e2e176a982096019c6c1bdd7d6c2ab5def663387bba8"} Oct 30 18:23:05.082426 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:23:05.082389 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:23:05.385962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:05.385914 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:05.429318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:05.429279 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d286b388-ca68-4bda-b76d-07d6f362b101-kube-api-access-k48sz\" (UniqueName: \"kubernetes.io/projected/d286b388-ca68-4bda-b76d-07d6f362b101-kube-api-access-k48sz\") pod \"d286b388-ca68-4bda-b76d-07d6f362b101\" (UID: \"d286b388-ca68-4bda-b76d-07d6f362b101\") " Oct 30 18:23:05.429717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:05.429407 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-xks-ca\") pod \"d286b388-ca68-4bda-b76d-07d6f362b101\" (UID: \"d286b388-ca68-4bda-b76d-07d6f362b101\") " Oct 30 18:23:05.429996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:05.429966 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-openshift-ca\") pod \"d286b388-ca68-4bda-b76d-07d6f362b101\" (UID: \"d286b388-ca68-4bda-b76d-07d6f362b101\") " Oct 30 18:23:05.430683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:05.430640 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-openshift-ca" pod "d286b388-ca68-4bda-b76d-07d6f362b101" (UID: "d286b388-ca68-4bda-b76d-07d6f362b101"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:05.431060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:05.431038 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:05.433478 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:05.433444 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d286b388-ca68-4bda-b76d-07d6f362b101-kube-api-access-k48sz" pod "d286b388-ca68-4bda-b76d-07d6f362b101" (UID: "d286b388-ca68-4bda-b76d-07d6f362b101"). InnerVolumeSpecName "kube-api-access-k48sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:05.433719 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:05.433691 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-xks-ca" pod "d286b388-ca68-4bda-b76d-07d6f362b101" (UID: "d286b388-ca68-4bda-b76d-07d6f362b101"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:05.532613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:05.532571 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k48sz\" (UniqueName: \"kubernetes.io/projected/d286b388-ca68-4bda-b76d-07d6f362b101-kube-api-access-k48sz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:05.532613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:05.532608 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d286b388-ca68-4bda-b76d-07d6f362b101-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:06.177866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:06.177839 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" Oct 30 18:23:06.178096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:06.177867 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5" event={"ID":"d286b388-ca68-4bda-b76d-07d6f362b101","Type":"ContainerDied","Data":"3c6900ab86803a83aafdbe863d8ba04c6926b25cf0b5ec0e8cd6136a2aa97523"} Oct 30 18:23:06.178096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:06.177922 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c6900ab86803a83aafdbe863d8ba04c6926b25cf0b5ec0e8cd6136a2aa97523" Oct 30 18:23:07.953433 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:07.953377 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" podUID="e0a90af2-5579-450a-9597-513a50296cfe" containerName="payload-pre-processing" containerID="cri-o://877031e92db6aa3ca332d00d52421f5c336fa4f57cbf8fa0a1c4661b25523bfb" gracePeriod=25 Oct 30 18:23:08.212022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:08.211102 4883 generic.go:356] "Generic (PLEG): container finished" podID="e0a90af2-5579-450a-9597-513a50296cfe" containerID="877031e92db6aa3ca332d00d52421f5c336fa4f57cbf8fa0a1c4661b25523bfb" exitCode=0 Oct 30 18:23:08.212022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:08.211364 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" event={"ID":"e0a90af2-5579-450a-9597-513a50296cfe","Type":"ContainerDied","Data":"877031e92db6aa3ca332d00d52421f5c336fa4f57cbf8fa0a1c4661b25523bfb"} Oct 30 18:23:08.261853 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:08.261817 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" Oct 30 18:23:08.373563 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:08.373528 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a90af2-5579-450a-9597-513a50296cfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a90af2-5579-450a-9597-513a50296cfe-plugins-config-volume\") pod \"e0a90af2-5579-450a-9597-513a50296cfe\" (UID: \"e0a90af2-5579-450a-9597-513a50296cfe\") " Oct 30 18:23:08.373757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:08.373637 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a90af2-5579-450a-9597-513a50296cfe-kube-api-access-4qbtn\" (UniqueName: \"kubernetes.io/projected/e0a90af2-5579-450a-9597-513a50296cfe-kube-api-access-4qbtn\") pod \"e0a90af2-5579-450a-9597-513a50296cfe\" (UID: \"e0a90af2-5579-450a-9597-513a50296cfe\") " Oct 30 18:23:08.373979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:08.373907 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a90af2-5579-450a-9597-513a50296cfe-plugins-config-volume" pod "e0a90af2-5579-450a-9597-513a50296cfe" (UID: "e0a90af2-5579-450a-9597-513a50296cfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:08.374126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:08.374109 4883 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a90af2-5579-450a-9597-513a50296cfe-plugins-config-volume\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:08.375996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:08.375965 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a90af2-5579-450a-9597-513a50296cfe-kube-api-access-4qbtn" pod "e0a90af2-5579-450a-9597-513a50296cfe" (UID: "e0a90af2-5579-450a-9597-513a50296cfe"). InnerVolumeSpecName "kube-api-access-4qbtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:08.475406 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:08.475309 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qbtn\" (UniqueName: \"kubernetes.io/projected/e0a90af2-5579-450a-9597-513a50296cfe-kube-api-access-4qbtn\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:09.222470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:09.222427 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" event={"ID":"e0a90af2-5579-450a-9597-513a50296cfe","Type":"ContainerDied","Data":"601c178881cf6953c7167798fee0360d45ec971a02d719d0b41fb537230d437f"} Oct 30 18:23:09.223523 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:09.222511 4883 scope.go:122] "RemoveContainer" containerID="877031e92db6aa3ca332d00d52421f5c336fa4f57cbf8fa0a1c4661b25523bfb" Oct 30 18:23:09.223523 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:09.222556 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt" Oct 30 18:23:09.285331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:09.285294 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt"] Oct 30 18:23:09.299018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:09.298907 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e0bc49-7b47578bc-pwswt"] Oct 30 18:23:11.179479 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:11.179445 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a90af2-5579-450a-9597-513a50296cfe" path="/var/lib/kubelet/pods/e0a90af2-5579-450a-9597-513a50296cfe/volumes" Oct 30 18:23:24.960485 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:24.960450 4883 scope.go:122] "RemoveContainer" containerID="ec10e7c8d9c8f6b93a2add458771fa11e7f81c95f3898217bbf40569d3d6411a" Oct 30 18:23:25.034302 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:25.034274 4883 scope.go:122] "RemoveContainer" containerID="caa978a6dc036a1188809e7d6bf7113b4e537f7a7a5031587176b6aa96a41ebd" Oct 30 18:23:25.046449 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:25.046427 4883 scope.go:122] "RemoveContainer" containerID="d067498ae874f2ae3bbb607c512a09826f6b9e9cbc419577a8ad42631bdfc405" Oct 30 18:23:25.184649 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:25.184582 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5"] Oct 30 18:23:25.221132 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:25.221024 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-394a31-4psr5"] Oct 30 18:23:27.180612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:27.180573 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d286b388-ca68-4bda-b76d-07d6f362b101" path="/var/lib/kubelet/pods/d286b388-ca68-4bda-b76d-07d6f362b101/volumes" Oct 30 18:23:30.785252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:30.785209 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5"] Oct 30 18:23:30.796343 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:30.796306 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e0bc49-xcxn5"] Oct 30 18:23:31.192172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:31.192132 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b1c58a-b359-4f57-a141-eeb36c130d4e" path="/var/lib/kubelet/pods/53b1c58a-b359-4f57-a141-eeb36c130d4e/volumes" Oct 30 18:23:35.087090 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:23:35.087054 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:23:38.554183 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:38.554061 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x"] Oct 30 18:23:38.555069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:38.554993 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" containerID="cri-o://687cc4202164770f159edab241dcd7f4bfa8922fe437583daa3d17fbe31d59b4" gracePeriod=30 Oct 30 18:23:40.151110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:40.151055 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.163532 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:40.163477 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.307102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:40.307049 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.451045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:40.441182 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.519964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:40.476100 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.581841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:40.580258 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.666963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:40.663035 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.804159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:40.804043 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:40.804766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:40.804734 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:23:40.842106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:40.842045 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:41.070244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.070139 4883 patch_prober.go:29] interesting pod/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:23:41.070517 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.070485 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:23:41.174429 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.171055 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:41.174429 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.171221 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:41.174429 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.171544 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:23:41.174429 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.172364 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:23:41.227139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.227083 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:41.464684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.460126 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:41.515988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.488273 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:23:41.658207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.655020 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:23:41.658207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.655106 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:23:41.658207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.655144 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:23:41.764024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.676334 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:23:41.809963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.691857 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:23:41.809963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.809598 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:23:41.828181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.809104 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:41.854157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.828479 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:23:41.888634 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.888593 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:41.889994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.889957 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:41.979968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:41.909993 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:23:42.106051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:42.102369 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:23:42.191086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:42.191029 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:42.207074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:42.191759 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:23:42.207074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:42.193065 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:23:42.232180 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:42.232123 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:23:42.267566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:42.267530 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:23:42.289612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:42.289539 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:42.500735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:42.500638 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6"] Oct 30 18:23:42.504314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:42.504263 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" containerID="cri-o://f9801000bdc104610306a9af1cead8766dac7d1736f30d196413925fe2cf9238" gracePeriod=30 Oct 30 18:23:42.583645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:42.583592 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:42.813199 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:42.813093 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:42.853341 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:42.852601 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:23:43.063561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:43.063471 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:23:43.240959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:43.237068 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:43.275341 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:43.275295 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:43.276596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:43.276565 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:43.282423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:43.282384 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:43.839987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:43.839415 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:23:44.112491 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.112435 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:44.213959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.213898 4883 generic.go:356] "Generic (PLEG): container finished" podID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerID="687cc4202164770f159edab241dcd7f4bfa8922fe437583daa3d17fbe31d59b4" exitCode=0 Oct 30 18:23:44.214535 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.214507 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" event={"ID":"32faa9f8-f224-4587-b81b-7ea9988fe4c6","Type":"ContainerDied","Data":"687cc4202164770f159edab241dcd7f4bfa8922fe437583daa3d17fbe31d59b4"} Oct 30 18:23:44.214799 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.214781 4883 scope.go:122] "RemoveContainer" containerID="1f4bbe2ccfb0e2b38056c7750ee695ada94847b2aed15f44c1ee94dbafd7a49f" Oct 30 18:23:44.253196 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.246234 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:44.253196 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.247067 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:23:44.282207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.276432 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:44.282207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.277089 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:23:44.282469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.282226 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:44.282770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.282732 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:44.290336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.290287 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:44.520338 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.520302 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:23:44.654029 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.648128 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:23:44.680960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.677223 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:44.868961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.844351 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:23:44.869422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.869373 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:44.883080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:44.845694 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:23:45.135963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.122490 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:45.135963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.123052 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:23:45.389852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.389753 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:23:45.400337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.400278 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:45.524431 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.425155 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:23:45.525059 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.425199 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:45.525478 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.426098 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:23:45.602260 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.602216 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:45.688021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.667636 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:23:45.724556 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.724525 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:23:45.724987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.677819 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:45.743132 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:23:45.743083 4883 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32faa9f8_f224_4587_b81b_7ea9988fe4c6.slice/crio-conmon-687cc4202164770f159edab241dcd7f4bfa8922fe437583daa3d17fbe31d59b4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32faa9f8_f224_4587_b81b_7ea9988fe4c6.slice/crio-687cc4202164770f159edab241dcd7f4bfa8922fe437583daa3d17fbe31d59b4.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:23:45.777969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.757965 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:23:45.851249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.850520 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:45.878966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.872740 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:23:45.961186 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.884856 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:46.040688 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.885187 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:23:46.041039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:45.985873 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:23:46.048971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.045001 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:23:46.218147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.218045 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:23:46.258244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.258197 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:23:46.536021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.532342 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:46.545125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.545061 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:23:46.556587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.556542 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:23:46.563682 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.558856 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-envoy\") pod \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " Oct 30 18:23:46.564018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.563995 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istiod-ca-cert\") pod \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " Oct 30 18:23:46.564215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.564194 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-credential-socket\") pod \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " Oct 30 18:23:46.576959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.564386 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-podinfo\") pod \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " Oct 30 18:23:46.577350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.577319 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-certs\") pod \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " Oct 30 18:23:46.577617 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.577598 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-token\" (UniqueName: \"kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-token\") pod \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " Oct 30 18:23:46.578082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.578056 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-socket\") pod \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " Oct 30 18:23:46.578262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.578243 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-kube-api-access-5v2xv\" (UniqueName: \"kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-kube-api-access-5v2xv\") pod \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " Oct 30 18:23:46.578423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.578406 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-data\") pod \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\" (UID: \"32faa9f8-f224-4587-b81b-7ea9988fe4c6\") " Oct 30 18:23:46.580133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.580107 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-certs" pod "32faa9f8-f224-4587-b81b-7ea9988fe4c6" (UID: "32faa9f8-f224-4587-b81b-7ea9988fe4c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.601957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.601897 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istiod-ca-cert" pod "32faa9f8-f224-4587-b81b-7ea9988fe4c6" (UID: "32faa9f8-f224-4587-b81b-7ea9988fe4c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:46.603440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.603411 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-credential-socket" pod "32faa9f8-f224-4587-b81b-7ea9988fe4c6" (UID: "32faa9f8-f224-4587-b81b-7ea9988fe4c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.604956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.604913 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-socket" pod "32faa9f8-f224-4587-b81b-7ea9988fe4c6" (UID: "32faa9f8-f224-4587-b81b-7ea9988fe4c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.605344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.605322 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-data" pod "32faa9f8-f224-4587-b81b-7ea9988fe4c6" (UID: "32faa9f8-f224-4587-b81b-7ea9988fe4c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.605821 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.605791 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:46.738195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.687566 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.738195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.687609 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.738195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.687624 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.738195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.687638 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.738195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.687651 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.738880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.738843 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:23:46.750999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.750345 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-token" pod "32faa9f8-f224-4587-b81b-7ea9988fe4c6" (UID: "32faa9f8-f224-4587-b81b-7ea9988fe4c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.750999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.750918 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-envoy" pod "32faa9f8-f224-4587-b81b-7ea9988fe4c6" (UID: "32faa9f8-f224-4587-b81b-7ea9988fe4c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.775995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.774659 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-kube-api-access-5v2xv" pod "32faa9f8-f224-4587-b81b-7ea9988fe4c6" (UID: "32faa9f8-f224-4587-b81b-7ea9988fe4c6"). InnerVolumeSpecName "kube-api-access-5v2xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.777094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.777058 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:46.824551 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.793191 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.824806 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.824781 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v2xv\" (UniqueName: \"kubernetes.io/projected/32faa9f8-f224-4587-b81b-7ea9988fe4c6-kube-api-access-5v2xv\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.824954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.824919 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.858977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.845972 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-podinfo" pod "32faa9f8-f224-4587-b81b-7ea9988fe4c6" (UID: "32faa9f8-f224-4587-b81b-7ea9988fe4c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:46.930541 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.918425 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded" Oct 30 18:23:47.210879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.940676 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.212892 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.941091 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:23:47.213409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.941355 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.218975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:46.991461 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32faa9f8-f224-4587-b81b-7ea9988fe4c6-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.228708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.055154 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:23:47.229387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.156222 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" event={"ID":"32faa9f8-f224-4587-b81b-7ea9988fe4c6","Type":"ContainerDied","Data":"fc1a5e57ba4b2443d4f32efec7cae09c747d2707dea184420d33f0741a84ad5a"} Oct 30 18:23:47.243276 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.243236 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:23:47.243951 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.156348 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x" Oct 30 18:23:47.244216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.243412 4883 scope.go:122] "RemoveContainer" containerID="687cc4202164770f159edab241dcd7f4bfa8922fe437583daa3d17fbe31d59b4" Oct 30 18:23:47.257498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.257433 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.259162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.259116 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.551281 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.547124 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.574628 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.565065 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.574628 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.565205 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.574628 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.565435 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.638998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.626602 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.638998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.627056 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:23:47.783998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.740138 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:47.862082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.850404 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x"] Oct 30 18:23:47.919022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:47.906199 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e0bc49-openshift-default-664fcbfd9-j2n7x"] Oct 30 18:23:48.318620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.250141 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.318620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.252223 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:23:48.319540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.252255 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:23:48.320044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.252276 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:23:48.320614 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.279917 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Oct 30 18:23:48.320881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.279983 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.321236 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.280012 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.323472 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.323451 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:23:48.323671 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.323655 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:23:48.365373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.365325 4883 generic.go:356] "Generic (PLEG): container finished" podID="4141ed82-3540-42d3-9346-368784ca91c0" containerID="f9801000bdc104610306a9af1cead8766dac7d1736f30d196413925fe2cf9238" exitCode=0 Oct 30 18:23:48.365845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.365815 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" event={"ID":"4141ed82-3540-42d3-9346-368784ca91c0","Type":"ContainerDied","Data":"f9801000bdc104610306a9af1cead8766dac7d1736f30d196413925fe2cf9238"} Oct 30 18:23:48.366065 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.366046 4883 scope.go:122] "RemoveContainer" containerID="1470e8d33a92d55ef75b0ef800f94d0951bd36aa76528c7f3e89e12eeed9901f" Oct 30 18:23:48.515032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.488902 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.562305 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.551376 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.723959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.723303 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:23:48.744175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.728440 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.744175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.728857 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:23:48.744175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.730141 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.744175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.732691 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:48.769702 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.769598 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:23:48.978386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:48.978353 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:23:49.051056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.031161 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:49.197530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.159237 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:23:49.283398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.283350 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:49.330788 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.330163 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:49.345472 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.342908 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:23:49.435547 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.434715 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:49.445127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.442692 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:49.452983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.452497 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:49.531381 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.531337 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" path="/var/lib/kubelet/pods/32faa9f8-f224-4587-b81b-7ea9988fe4c6/volumes" Oct 30 18:23:49.532194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.532171 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:23:49.536007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.534463 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:23:49.544038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.542789 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:23:49.572987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.569532 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" event={"ID":"4141ed82-3540-42d3-9346-368784ca91c0","Type":"ContainerDied","Data":"b970e1cde5b684257274423fb720f98ec42593b689eb3719999a852a82e15e59"} Oct 30 18:23:49.572987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.569635 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b970e1cde5b684257274423fb720f98ec42593b689eb3719999a852a82e15e59" Oct 30 18:23:49.572987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.569750 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:23:49.572987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.569825 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:23:49.572987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.569881 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:23:49.572987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.569950 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:23:49.572987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.570001 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:23:49.572987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.570057 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:23:49.620854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.620812 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:23:49.635897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.635863 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:23:49.662201 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.661456 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-kube-api-access-vv2sh\" (UniqueName: \"kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-kube-api-access-vv2sh\") pod \"4141ed82-3540-42d3-9346-368784ca91c0\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " Oct 30 18:23:49.662505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.662481 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-credential-socket\") pod \"4141ed82-3540-42d3-9346-368784ca91c0\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " Oct 30 18:23:49.662993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.662796 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-credential-socket" pod "4141ed82-3540-42d3-9346-368784ca91c0" (UID: "4141ed82-3540-42d3-9346-368784ca91c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:49.663336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.663313 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-envoy\") pod \"4141ed82-3540-42d3-9346-368784ca91c0\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " Oct 30 18:23:49.663483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.663465 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4141ed82-3540-42d3-9346-368784ca91c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4141ed82-3540-42d3-9346-368784ca91c0-istio-podinfo\") pod \"4141ed82-3540-42d3-9346-368784ca91c0\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " Oct 30 18:23:49.663582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.663566 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-certs\") pod \"4141ed82-3540-42d3-9346-368784ca91c0\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " Oct 30 18:23:49.663681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.663660 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-data\") pod \"4141ed82-3540-42d3-9346-368784ca91c0\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " Oct 30 18:23:49.663741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.663709 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-socket\") pod \"4141ed82-3540-42d3-9346-368784ca91c0\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " Oct 30 18:23:49.663801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.663750 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-istio-token\" (UniqueName: \"kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-istio-token\") pod \"4141ed82-3540-42d3-9346-368784ca91c0\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " Oct 30 18:23:49.663854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.663799 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4141ed82-3540-42d3-9346-368784ca91c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4141ed82-3540-42d3-9346-368784ca91c0-istiod-ca-cert\") pod \"4141ed82-3540-42d3-9346-368784ca91c0\" (UID: \"4141ed82-3540-42d3-9346-368784ca91c0\") " Oct 30 18:23:49.665170 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.665115 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-kube-api-access-vv2sh" pod "4141ed82-3540-42d3-9346-368784ca91c0" (UID: "4141ed82-3540-42d3-9346-368784ca91c0"). InnerVolumeSpecName "kube-api-access-vv2sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:49.665301 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.665279 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4141ed82-3540-42d3-9346-368784ca91c0-istiod-ca-cert" pod "4141ed82-3540-42d3-9346-368784ca91c0" (UID: "4141ed82-3540-42d3-9346-368784ca91c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:49.665457 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.665432 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv2sh\" (UniqueName: \"kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-kube-api-access-vv2sh\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:49.665529 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.665466 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:49.666341 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.666318 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-data" pod "4141ed82-3540-42d3-9346-368784ca91c0" (UID: "4141ed82-3540-42d3-9346-368784ca91c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:49.667496 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.667462 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-socket" pod "4141ed82-3540-42d3-9346-368784ca91c0" (UID: "4141ed82-3540-42d3-9346-368784ca91c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:49.668508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.667669 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-certs" pod "4141ed82-3540-42d3-9346-368784ca91c0" (UID: "4141ed82-3540-42d3-9346-368784ca91c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:49.669832 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.669807 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-envoy" pod "4141ed82-3540-42d3-9346-368784ca91c0" (UID: "4141ed82-3540-42d3-9346-368784ca91c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:49.672351 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.672293 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4141ed82-3540-42d3-9346-368784ca91c0-istio-podinfo" pod "4141ed82-3540-42d3-9346-368784ca91c0" (UID: "4141ed82-3540-42d3-9346-368784ca91c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:49.672682 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.672641 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-istio-token" pod "4141ed82-3540-42d3-9346-368784ca91c0" (UID: "4141ed82-3540-42d3-9346-368784ca91c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:49.766798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.766751 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4141ed82-3540-42d3-9346-368784ca91c0-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:49.766798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.766792 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:49.767113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.766808 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:49.767113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.766821 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:49.767113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.766835 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4141ed82-3540-42d3-9346-368784ca91c0-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:49.767113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.766850 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4141ed82-3540-42d3-9346-368784ca91c0-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:49.767113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:49.766862 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4141ed82-3540-42d3-9346-368784ca91c0-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:23:50.522172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:50.522135 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6" Oct 30 18:23:50.582020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:50.581981 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6"] Oct 30 18:23:50.593178 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:50.593142 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-394a31-openshift-default-7c846788c5-sgpx6"] Oct 30 18:23:51.179321 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:23:51.179285 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4141ed82-3540-42d3-9346-368784ca91c0" path="/var/lib/kubelet/pods/4141ed82-3540-42d3-9346-368784ca91c0/volumes" Oct 30 18:24:05.081354 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:24:05.081312 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:24:25.145562 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:25.145527 4883 scope.go:122] "RemoveContainer" containerID="b25631f09afad19b094f61800fd9f407eafaf5bdbef1deee1ae02bef625b21c0" Oct 30 18:24:25.156097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:25.156070 4883 scope.go:122] "RemoveContainer" containerID="9b5d5629c4d182ae445863c458bd60bec877444b8588294440c94b72d965e7aa" Oct 30 18:24:25.168310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:25.168275 4883 scope.go:122] "RemoveContainer" containerID="1f68c50851b48b2b71ee4f4a3120ff108151a04410b82330ae329bfbdbcb2284" Oct 30 18:24:26.354604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.354562 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj"] Oct 30 18:24:26.355126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355111 4883 cpu_manager.go:414] "containerMap: removing container" podUID="d286b388-ca68-4bda-b76d-07d6f362b101" containerName="revoke-keys" Oct 30 18:24:26.355174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355127 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d286b388-ca68-4bda-b76d-07d6f362b101" containerName="revoke-keys" Oct 30 18:24:26.355174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355137 4883 cpu_manager.go:414] "containerMap: removing container" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" Oct 30 18:24:26.355174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355143 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" Oct 30 18:24:26.355174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355167 4883 cpu_manager.go:414] "containerMap: removing container" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" Oct 30 18:24:26.355174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355173 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" Oct 30 18:24:26.355344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355185 4883 cpu_manager.go:414] "containerMap: removing container" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" Oct 30 18:24:26.355344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355192 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" Oct 30 18:24:26.355344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355199 4883 cpu_manager.go:414] "containerMap: removing container" podUID="e0a90af2-5579-450a-9597-513a50296cfe" containerName="payload-pre-processing" Oct 30 18:24:26.355344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355204 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a90af2-5579-450a-9597-513a50296cfe" containerName="payload-pre-processing" Oct 30 18:24:26.355344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355298 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="d286b388-ca68-4bda-b76d-07d6f362b101" containerName="revoke-keys" Oct 30 18:24:26.355344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355306 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" Oct 30 18:24:26.355344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355315 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" Oct 30 18:24:26.355344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355321 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="4141ed82-3540-42d3-9346-368784ca91c0" containerName="istio-proxy" Oct 30 18:24:26.355344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355327 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a90af2-5579-450a-9597-513a50296cfe" containerName="payload-pre-processing" Oct 30 18:24:26.355344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355336 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" Oct 30 18:24:26.355344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.355341 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a0c1ed-a134-4e64-ad6f-1a5d3a7cdd83" containerName="istio-proxy" Oct 30 18:24:26.360927 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.360897 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:26.379499 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.379459 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj"] Oct 30 18:24:26.389366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.389334 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:24:26.389920 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.389879 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:24:26.390084 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.390044 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:24:26.390147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.390134 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:24:26.463997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.452080 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj\" (UID: \"b305597e-d6a3-408d-ab17-0904638495b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:26.463997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.452316 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj\" (UID: \"b305597e-d6a3-408d-ab17-0904638495b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:26.463997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.452394 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x799w\" (UniqueName: \"kubernetes.io/projected/b305597e-d6a3-408d-ab17-0904638495b9-kube-api-access-x799w\") pod \"maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj\" (UID: \"b305597e-d6a3-408d-ab17-0904638495b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:26.553359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.553316 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x799w\" (UniqueName: \"kubernetes.io/projected/b305597e-d6a3-408d-ab17-0904638495b9-kube-api-access-x799w\") pod \"maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj\" (UID: \"b305597e-d6a3-408d-ab17-0904638495b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:26.553568 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.553513 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj\" (UID: \"b305597e-d6a3-408d-ab17-0904638495b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:26.553776 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.553664 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj\" (UID: \"b305597e-d6a3-408d-ab17-0904638495b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:26.554441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.554417 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj\" (UID: \"b305597e-d6a3-408d-ab17-0904638495b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:26.559333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.559297 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj\" (UID: \"b305597e-d6a3-408d-ab17-0904638495b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:26.579499 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.579458 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x799w\" (UniqueName: \"kubernetes.io/projected/b305597e-d6a3-408d-ab17-0904638495b9-kube-api-access-x799w\") pod \"maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj\" (UID: \"b305597e-d6a3-408d-ab17-0904638495b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:26.676021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.675914 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:26.983414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:26.983383 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj"] Oct 30 18:24:26.985849 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:24:26.985822 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb305597e_d6a3_408d_ab17_0904638495b9.slice/crio-c683c34f64e89fd6fde8257a1fadba3ab8279cce2d2456309bbc0c4ab6551692 WatchSource:0}: Error finding container c683c34f64e89fd6fde8257a1fadba3ab8279cce2d2456309bbc0c4ab6551692: Status 404 returned error can't find the container with id c683c34f64e89fd6fde8257a1fadba3ab8279cce2d2456309bbc0c4ab6551692 Oct 30 18:24:27.902303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:27.902270 4883 generic.go:356] "Generic (PLEG): container finished" podID="b305597e-d6a3-408d-ab17-0904638495b9" containerID="b4529a30d4e0f37c55f731d2e0b7815f3fee16e21f5050a1ada09c02b08d355e" exitCode=0 Oct 30 18:24:27.902695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:27.902356 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" event={"ID":"b305597e-d6a3-408d-ab17-0904638495b9","Type":"ContainerDied","Data":"b4529a30d4e0f37c55f731d2e0b7815f3fee16e21f5050a1ada09c02b08d355e"} Oct 30 18:24:27.902695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:27.902412 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" event={"ID":"b305597e-d6a3-408d-ab17-0904638495b9","Type":"ContainerStarted","Data":"c683c34f64e89fd6fde8257a1fadba3ab8279cce2d2456309bbc0c4ab6551692"} Oct 30 18:24:29.080870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.080828 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:29.194857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.194763 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b305597e-d6a3-408d-ab17-0904638495b9-kube-api-access-x799w\" (UniqueName: \"kubernetes.io/projected/b305597e-d6a3-408d-ab17-0904638495b9-kube-api-access-x799w\") pod \"b305597e-d6a3-408d-ab17-0904638495b9\" (UID: \"b305597e-d6a3-408d-ab17-0904638495b9\") " Oct 30 18:24:29.194857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.194809 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-openshift-ca\") pod \"b305597e-d6a3-408d-ab17-0904638495b9\" (UID: \"b305597e-d6a3-408d-ab17-0904638495b9\") " Oct 30 18:24:29.195125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.195039 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-xks-ca\") pod \"b305597e-d6a3-408d-ab17-0904638495b9\" (UID: \"b305597e-d6a3-408d-ab17-0904638495b9\") " Oct 30 18:24:29.195253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.195228 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-openshift-ca" pod "b305597e-d6a3-408d-ab17-0904638495b9" (UID: "b305597e-d6a3-408d-ab17-0904638495b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:29.195534 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.195517 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:24:29.197234 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.197207 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-xks-ca" pod "b305597e-d6a3-408d-ab17-0904638495b9" (UID: "b305597e-d6a3-408d-ab17-0904638495b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:29.197391 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.197375 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b305597e-d6a3-408d-ab17-0904638495b9-kube-api-access-x799w" pod "b305597e-d6a3-408d-ab17-0904638495b9" (UID: "b305597e-d6a3-408d-ab17-0904638495b9"). InnerVolumeSpecName "kube-api-access-x799w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:29.296551 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.296515 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b305597e-d6a3-408d-ab17-0904638495b9-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:24:29.296551 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.296547 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x799w\" (UniqueName: \"kubernetes.io/projected/b305597e-d6a3-408d-ab17-0904638495b9-kube-api-access-x799w\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:24:29.918306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.918261 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" event={"ID":"b305597e-d6a3-408d-ab17-0904638495b9","Type":"ContainerDied","Data":"c683c34f64e89fd6fde8257a1fadba3ab8279cce2d2456309bbc0c4ab6551692"} Oct 30 18:24:29.918492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.918308 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj" Oct 30 18:24:29.918492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:29.918338 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c683c34f64e89fd6fde8257a1fadba3ab8279cce2d2456309bbc0c4ab6551692" Oct 30 18:24:35.081490 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:24:35.081396 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:24:35.971524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:35.971488 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk"] Oct 30 18:24:35.972175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:35.972157 4883 cpu_manager.go:414] "containerMap: removing container" podUID="b305597e-d6a3-408d-ab17-0904638495b9" containerName="revoke-keys" Oct 30 18:24:35.972239 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:35.972178 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b305597e-d6a3-408d-ab17-0904638495b9" containerName="revoke-keys" Oct 30 18:24:35.972279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:35.972237 4883 cpu_manager.go:414] "containerMap: removing container" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" Oct 30 18:24:35.972279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:35.972245 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32faa9f8-f224-4587-b81b-7ea9988fe4c6" containerName="istio-proxy" Oct 30 18:24:35.972376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:35.972363 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="b305597e-d6a3-408d-ab17-0904638495b9" containerName="revoke-keys" Oct 30 18:24:35.976401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:35.976372 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:35.996158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:35.996124 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk"] Oct 30 18:24:36.007980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.005696 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:24:36.007980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.005809 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:24:36.007980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.005821 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:24:36.007980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.005967 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:24:36.107998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.107917 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk\" (UID: \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:36.108402 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.108067 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk\" (UID: \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:36.108402 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.108239 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8wk\" (UniqueName: \"kubernetes.io/projected/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-kube-api-access-6c8wk\") pod \"maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk\" (UID: \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:36.209919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.209877 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk\" (UID: \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:36.210131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.210035 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8wk\" (UniqueName: \"kubernetes.io/projected/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-kube-api-access-6c8wk\") pod \"maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk\" (UID: \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:36.210185 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.210168 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk\" (UID: \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:36.210590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.210568 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk\" (UID: \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:36.212476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.212452 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk\" (UID: \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:36.235322 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.235246 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8wk\" (UniqueName: \"kubernetes.io/projected/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-kube-api-access-6c8wk\") pod \"maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk\" (UID: \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:36.294177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.294141 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:36.609515 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.609485 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk"] Oct 30 18:24:36.610758 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:24:36.610727 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d004ac5_9dd5_41fa_a158_d198dc4dbadc.slice/crio-e437b2a6356f3b8b3304104074afd0aa3a7cccadad1432cf9ae6da1c754fe4b6 WatchSource:0}: Error finding container e437b2a6356f3b8b3304104074afd0aa3a7cccadad1432cf9ae6da1c754fe4b6: Status 404 returned error can't find the container with id e437b2a6356f3b8b3304104074afd0aa3a7cccadad1432cf9ae6da1c754fe4b6 Oct 30 18:24:36.971847 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:36.971806 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" event={"ID":"1d004ac5-9dd5-41fa-a158-d198dc4dbadc","Type":"ContainerStarted","Data":"e437b2a6356f3b8b3304104074afd0aa3a7cccadad1432cf9ae6da1c754fe4b6"} Oct 30 18:24:37.981140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:37.981103 4883 generic.go:356] "Generic (PLEG): container finished" podID="1d004ac5-9dd5-41fa-a158-d198dc4dbadc" containerID="6591233accedeef2c7961d264812c151d16da86e94a7195e67494bf06f2c051a" exitCode=0 Oct 30 18:24:37.981140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:37.981152 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" event={"ID":"1d004ac5-9dd5-41fa-a158-d198dc4dbadc","Type":"ContainerDied","Data":"6591233accedeef2c7961d264812c151d16da86e94a7195e67494bf06f2c051a"} Oct 30 18:24:39.264055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:39.264023 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:39.365767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:39.365727 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-xks-ca\") pod \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\" (UID: \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\") " Oct 30 18:24:39.365999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:39.365902 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-kube-api-access-6c8wk\" (UniqueName: \"kubernetes.io/projected/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-kube-api-access-6c8wk\") pod \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\" (UID: \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\") " Oct 30 18:24:39.366140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:39.366110 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-openshift-ca\") pod \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\" (UID: \"1d004ac5-9dd5-41fa-a158-d198dc4dbadc\") " Oct 30 18:24:39.366506 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:39.366477 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-openshift-ca" pod "1d004ac5-9dd5-41fa-a158-d198dc4dbadc" (UID: "1d004ac5-9dd5-41fa-a158-d198dc4dbadc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:39.366917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:39.366829 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:24:39.368256 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:39.368225 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-xks-ca" pod "1d004ac5-9dd5-41fa-a158-d198dc4dbadc" (UID: "1d004ac5-9dd5-41fa-a158-d198dc4dbadc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:39.368461 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:39.368434 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-kube-api-access-6c8wk" pod "1d004ac5-9dd5-41fa-a158-d198dc4dbadc" (UID: "1d004ac5-9dd5-41fa-a158-d198dc4dbadc"). InnerVolumeSpecName "kube-api-access-6c8wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:39.470381 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:39.470338 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:24:39.470381 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:39.470378 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c8wk\" (UniqueName: \"kubernetes.io/projected/1d004ac5-9dd5-41fa-a158-d198dc4dbadc-kube-api-access-6c8wk\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:24:40.051259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:40.051215 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" event={"ID":"1d004ac5-9dd5-41fa-a158-d198dc4dbadc","Type":"ContainerDied","Data":"e437b2a6356f3b8b3304104074afd0aa3a7cccadad1432cf9ae6da1c754fe4b6"} Oct 30 18:24:40.051259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:40.051251 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk" Oct 30 18:24:40.051259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:40.051264 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e437b2a6356f3b8b3304104074afd0aa3a7cccadad1432cf9ae6da1c754fe4b6" Oct 30 18:24:42.581909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.581335 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks"] Oct 30 18:24:42.582468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.582197 4883 cpu_manager.go:414] "containerMap: removing container" podUID="1d004ac5-9dd5-41fa-a158-d198dc4dbadc" containerName="revoke-keys" Oct 30 18:24:42.582468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.582215 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d004ac5-9dd5-41fa-a158-d198dc4dbadc" containerName="revoke-keys" Oct 30 18:24:42.582468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.582353 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d004ac5-9dd5-41fa-a158-d198dc4dbadc" containerName="revoke-keys" Oct 30 18:24:42.586352 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.586326 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.610161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.610125 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-42fad7-openshift-default-dockercfg-vtrtw\"" Oct 30 18:24:42.621547 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.621507 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks"] Oct 30 18:24:42.723846 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.723804 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-token\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.723846 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.723845 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549a0999-9c9c-414a-ab9b-8574a8660dbf-istiod-ca-cert\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.724146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.724043 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vm6\" (UniqueName: \"kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-kube-api-access-d4vm6\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.724146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.724107 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-podinfo\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.724229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.724151 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-credential-socket\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.724229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.724220 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-envoy\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.724348 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.724252 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-data\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.724405 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.724350 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-socket\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.724454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.724441 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-certs\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.831849 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.831802 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-token\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.832892 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.832808 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549a0999-9c9c-414a-ab9b-8574a8660dbf-istiod-ca-cert\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.833897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.833785 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549a0999-9c9c-414a-ab9b-8574a8660dbf-istiod-ca-cert\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.834182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.834163 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vm6\" (UniqueName: \"kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-kube-api-access-d4vm6\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.834625 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.834606 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-podinfo\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.835311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.835289 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-credential-socket\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.835870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.835852 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-envoy\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.836017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.836001 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-data\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.836177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.836161 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-socket\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.837286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.837267 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-certs\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.838019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.835798 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-credential-socket\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.838188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.836812 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-socket\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.838320 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.837646 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-certs\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.838411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.836565 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-data\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.846561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.846528 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-envoy\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.866857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.866827 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vm6\" (UniqueName: \"kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-kube-api-access-d4vm6\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.879143 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.875605 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-podinfo\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.879332 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.879252 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-token\") pod \"e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:42.927961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:42.924092 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:43.285982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:43.285908 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks"] Oct 30 18:24:43.287632 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:24:43.287601 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549a0999_9c9c_414a_ab9b_8574a8660dbf.slice/crio-e4509e1e30f328f8c93a87acfbaae83c48e6c9e575c5750f821b750a991d15a2 WatchSource:0}: Error finding container e4509e1e30f328f8c93a87acfbaae83c48e6c9e575c5750f821b750a991d15a2: Status 404 returned error can't find the container with id e4509e1e30f328f8c93a87acfbaae83c48e6c9e575c5750f821b750a991d15a2 Oct 30 18:24:43.290865 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:43.290828 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:24:43.290997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:43.290904 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:24:43.290997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:43.290952 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:24:44.088450 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:44.088412 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" event={"ID":"549a0999-9c9c-414a-ab9b-8574a8660dbf","Type":"ContainerStarted","Data":"5785b57fb390aec729843b19a0fb7ef4211a37f267eea03f94a143b5d0a723b5"} Oct 30 18:24:44.088884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:44.088480 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" event={"ID":"549a0999-9c9c-414a-ab9b-8574a8660dbf","Type":"ContainerStarted","Data":"e4509e1e30f328f8c93a87acfbaae83c48e6c9e575c5750f821b750a991d15a2"} Oct 30 18:24:44.153481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:44.150841 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" podStartSLOduration=2.150820761 podStartE2EDuration="2.150820761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:44.149590476 +0000 UTC m=+1943.106594313" watchObservedRunningTime="2026-10-30 18:24:44.150820761 +0000 UTC m=+1943.107824597" Oct 30 18:24:44.925015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:44.924972 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:44.931054 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:44.931024 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:45.095887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:45.095857 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:45.096885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:45.096865 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:24:46.583807 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:46.583771 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj"] Oct 30 18:24:46.586897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:46.586876 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ef208a-2w6xj"] Oct 30 18:24:47.179109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:47.179070 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b305597e-d6a3-408d-ab17-0904638495b9" path="/var/lib/kubelet/pods/b305597e-d6a3-408d-ab17-0904638495b9/volumes" Oct 30 18:24:53.936676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:53.936633 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9"] Oct 30 18:24:53.942334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:53.942302 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:53.956703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:53.956594 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9"] Oct 30 18:24:53.975835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:53.975796 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:24:53.976085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:53.975962 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:24:53.976085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:53.976012 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:24:53.976687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:53.976664 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:24:54.041769 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:54.041732 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5dqm\" (UniqueName: \"kubernetes.io/projected/6883fb30-8ec0-4491-8c1f-368997e160bd-kube-api-access-p5dqm\") pod \"maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9\" (UID: \"6883fb30-8ec0-4491-8c1f-368997e160bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:54.041997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:54.041854 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9\" (UID: \"6883fb30-8ec0-4491-8c1f-368997e160bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:54.041997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:54.041901 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9\" (UID: \"6883fb30-8ec0-4491-8c1f-368997e160bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:54.143205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:54.143166 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5dqm\" (UniqueName: \"kubernetes.io/projected/6883fb30-8ec0-4491-8c1f-368997e160bd-kube-api-access-p5dqm\") pod \"maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9\" (UID: \"6883fb30-8ec0-4491-8c1f-368997e160bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:54.143366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:54.143323 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9\" (UID: \"6883fb30-8ec0-4491-8c1f-368997e160bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:54.143366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:54.143353 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9\" (UID: \"6883fb30-8ec0-4491-8c1f-368997e160bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:54.144163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:54.144139 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9\" (UID: \"6883fb30-8ec0-4491-8c1f-368997e160bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:54.145829 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:54.145808 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9\" (UID: \"6883fb30-8ec0-4491-8c1f-368997e160bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:54.168723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:54.168689 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5dqm\" (UniqueName: \"kubernetes.io/projected/6883fb30-8ec0-4491-8c1f-368997e160bd-kube-api-access-p5dqm\") pod \"maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9\" (UID: \"6883fb30-8ec0-4491-8c1f-368997e160bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:54.286621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:54.286528 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:54.593915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:54.593885 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9"] Oct 30 18:24:54.595677 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:24:54.595649 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6883fb30_8ec0_4491_8c1f_368997e160bd.slice/crio-4d137a359dd9c6e51794f6e27f3230399e9b52433bdb1252dd9fcd08984344b3 WatchSource:0}: Error finding container 4d137a359dd9c6e51794f6e27f3230399e9b52433bdb1252dd9fcd08984344b3: Status 404 returned error can't find the container with id 4d137a359dd9c6e51794f6e27f3230399e9b52433bdb1252dd9fcd08984344b3 Oct 30 18:24:55.178772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:55.178689 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" event={"ID":"6883fb30-8ec0-4491-8c1f-368997e160bd","Type":"ContainerStarted","Data":"45224494045f6e6e541ecb6fded352b87c2d1218f50208a5d67ff5f0c6f90b35"} Oct 30 18:24:55.178772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:55.178744 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" event={"ID":"6883fb30-8ec0-4491-8c1f-368997e160bd","Type":"ContainerStarted","Data":"4d137a359dd9c6e51794f6e27f3230399e9b52433bdb1252dd9fcd08984344b3"} Oct 30 18:24:55.220232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:55.220176 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" podStartSLOduration=2.220160797 podStartE2EDuration="2.220160797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:55.217649838 +0000 UTC m=+1954.174653677" watchObservedRunningTime="2026-10-30 18:24:55.220160797 +0000 UTC m=+1954.177164632" Oct 30 18:24:56.183397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:56.183360 4883 generic.go:356] "Generic (PLEG): container finished" podID="6883fb30-8ec0-4491-8c1f-368997e160bd" containerID="45224494045f6e6e541ecb6fded352b87c2d1218f50208a5d67ff5f0c6f90b35" exitCode=0 Oct 30 18:24:56.183796 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:56.183452 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" event={"ID":"6883fb30-8ec0-4491-8c1f-368997e160bd","Type":"ContainerDied","Data":"45224494045f6e6e541ecb6fded352b87c2d1218f50208a5d67ff5f0c6f90b35"} Oct 30 18:24:57.395060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:57.395029 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:57.507053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:57.495793 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6883fb30-8ec0-4491-8c1f-368997e160bd-kube-api-access-p5dqm\" (UniqueName: \"kubernetes.io/projected/6883fb30-8ec0-4491-8c1f-368997e160bd-kube-api-access-p5dqm\") pod \"6883fb30-8ec0-4491-8c1f-368997e160bd\" (UID: \"6883fb30-8ec0-4491-8c1f-368997e160bd\") " Oct 30 18:24:57.507053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:57.495853 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-openshift-ca\") pod \"6883fb30-8ec0-4491-8c1f-368997e160bd\" (UID: \"6883fb30-8ec0-4491-8c1f-368997e160bd\") " Oct 30 18:24:57.507053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:57.496049 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-xks-ca\") pod \"6883fb30-8ec0-4491-8c1f-368997e160bd\" (UID: \"6883fb30-8ec0-4491-8c1f-368997e160bd\") " Oct 30 18:24:57.663114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:57.663061 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-xks-ca" pod "6883fb30-8ec0-4491-8c1f-368997e160bd" (UID: "6883fb30-8ec0-4491-8c1f-368997e160bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:57.663660 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:57.663631 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-openshift-ca" pod "6883fb30-8ec0-4491-8c1f-368997e160bd" (UID: "6883fb30-8ec0-4491-8c1f-368997e160bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:57.675738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:57.675637 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6883fb30-8ec0-4491-8c1f-368997e160bd-kube-api-access-p5dqm" pod "6883fb30-8ec0-4491-8c1f-368997e160bd" (UID: "6883fb30-8ec0-4491-8c1f-368997e160bd"). InnerVolumeSpecName "kube-api-access-p5dqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:57.724738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:57.724699 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.724738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:57.724737 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5dqm\" (UniqueName: \"kubernetes.io/projected/6883fb30-8ec0-4491-8c1f-368997e160bd-kube-api-access-p5dqm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:24:57.724979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:57.724750 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6883fb30-8ec0-4491-8c1f-368997e160bd-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:24:58.202046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:58.202010 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" Oct 30 18:24:58.202270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:58.202033 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9" event={"ID":"6883fb30-8ec0-4491-8c1f-368997e160bd","Type":"ContainerDied","Data":"4d137a359dd9c6e51794f6e27f3230399e9b52433bdb1252dd9fcd08984344b3"} Oct 30 18:24:58.202270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:58.202100 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d137a359dd9c6e51794f6e27f3230399e9b52433bdb1252dd9fcd08984344b3" Oct 30 18:24:58.536115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:58.536012 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk"] Oct 30 18:24:58.539222 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:58.539190 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d42172-htjjk"] Oct 30 18:24:59.179551 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:24:59.179518 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d004ac5-9dd5-41fa-a158-d198dc4dbadc" path="/var/lib/kubelet/pods/1d004ac5-9dd5-41fa-a158-d198dc4dbadc/volumes" Oct 30 18:25:05.082699 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:25:05.082651 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:25:06.299527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.299485 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c"] Oct 30 18:25:06.300076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.300062 4883 cpu_manager.go:414] "containerMap: removing container" podUID="6883fb30-8ec0-4491-8c1f-368997e160bd" containerName="revoke-keys" Oct 30 18:25:06.300145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.300078 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6883fb30-8ec0-4491-8c1f-368997e160bd" containerName="revoke-keys" Oct 30 18:25:06.300251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.300229 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="6883fb30-8ec0-4491-8c1f-368997e160bd" containerName="revoke-keys" Oct 30 18:25:06.305144 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.305116 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:06.325572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.325536 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c"] Oct 30 18:25:06.336408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.336370 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:25:06.336628 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.336469 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:25:06.380228 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.380173 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6snlr\" (UniqueName: \"kubernetes.io/projected/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-kube-api-access-6snlr\") pod \"maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c\" (UID: \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:06.380527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.380507 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c\" (UID: \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:06.380600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.380560 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c\" (UID: \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:06.387264 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.387219 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:25:06.387456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.387334 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:25:06.482092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.482055 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c\" (UID: \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:06.482092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.482105 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c\" (UID: \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:06.482374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.482265 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snlr\" (UniqueName: \"kubernetes.io/projected/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-kube-api-access-6snlr\") pod \"maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c\" (UID: \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:06.482864 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.482833 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c\" (UID: \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:06.485053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.485029 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c\" (UID: \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:06.507503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.507462 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snlr\" (UniqueName: \"kubernetes.io/projected/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-kube-api-access-6snlr\") pod \"maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c\" (UID: \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:06.620156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.620113 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:06.929252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:06.929157 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c"] Oct 30 18:25:06.937815 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:25:06.937774 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b4f1fb_6f3c_47cc_bc0c_421bddc2bd77.slice/crio-d5906e40f03045e471113675aafbc1b419cec8df4be6b77fd34cc4f3e6b042ff WatchSource:0}: Error finding container d5906e40f03045e471113675aafbc1b419cec8df4be6b77fd34cc4f3e6b042ff: Status 404 returned error can't find the container with id d5906e40f03045e471113675aafbc1b419cec8df4be6b77fd34cc4f3e6b042ff Oct 30 18:25:07.270004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:07.269917 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" event={"ID":"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77","Type":"ContainerStarted","Data":"d5906e40f03045e471113675aafbc1b419cec8df4be6b77fd34cc4f3e6b042ff"} Oct 30 18:25:08.278397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:08.278359 4883 generic.go:356] "Generic (PLEG): container finished" podID="b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77" containerID="d10b6721b3bebde09fdc9ca1ee42e80815657a889a3e29b82e33d2f02ffa1aac" exitCode=0 Oct 30 18:25:08.278886 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:08.278449 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" event={"ID":"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77","Type":"ContainerDied","Data":"d10b6721b3bebde09fdc9ca1ee42e80815657a889a3e29b82e33d2f02ffa1aac"} Oct 30 18:25:09.457909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:09.457879 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:09.633361 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:09.633199 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-kube-api-access-6snlr\" (UniqueName: \"kubernetes.io/projected/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-kube-api-access-6snlr\") pod \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\" (UID: \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\") " Oct 30 18:25:09.634068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:09.634044 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-openshift-ca\") pod \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\" (UID: \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\") " Oct 30 18:25:09.634206 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:09.634096 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-xks-ca\") pod \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\" (UID: \"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77\") " Oct 30 18:25:09.634437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:09.634409 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-openshift-ca" pod "b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77" (UID: "b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:09.634676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:09.634658 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:09.635731 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:09.635708 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-kube-api-access-6snlr" pod "b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77" (UID: "b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77"). InnerVolumeSpecName "kube-api-access-6snlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:09.641739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:09.641698 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-xks-ca" pod "b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77" (UID: "b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:09.736044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:09.736002 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6snlr\" (UniqueName: \"kubernetes.io/projected/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-kube-api-access-6snlr\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:09.736044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:09.736044 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:10.297892 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:10.297796 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" event={"ID":"b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77","Type":"ContainerDied","Data":"d5906e40f03045e471113675aafbc1b419cec8df4be6b77fd34cc4f3e6b042ff"} Oct 30 18:25:10.297892 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:10.297821 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c" Oct 30 18:25:10.297892 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:10.297862 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5906e40f03045e471113675aafbc1b419cec8df4be6b77fd34cc4f3e6b042ff" Oct 30 18:25:12.477089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:12.477036 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:12.593116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:12.593067 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:12.831018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:12.819033 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.025164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:13.025094 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.198960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:13.198072 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.269644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:13.252040 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.462016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:13.461053 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.462231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:13.462119 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.489961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:13.488955 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.489961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:13.489537 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:25:13.549959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:13.546089 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:25:13.639403 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:13.596033 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:13.639403 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:13.596619 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:25:13.852430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:13.852376 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:25:13.853028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:13.852999 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:25:14.120789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.115099 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:14.120789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.115717 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:25:14.218971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.202122 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:14.272959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.219821 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:25:14.427015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.274470 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:25:14.900758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.334051 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:14.938049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.334092 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:14.979447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.458060 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:14.979708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.518652 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:14.983960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.518685 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:25:15.015468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.518722 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:15.016673 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.678676 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:25:15.024144 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.680204 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:25:15.036441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.869660 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:15.036441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:14.901866 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:25:15.036441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.030672 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:25:15.036441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.030738 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:25:15.036441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.030781 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:25:15.131960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.121033 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:15.224018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.222708 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:15.560215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.560113 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:15.560518 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.560486 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:15.680221 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.680163 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:15.681551 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.681514 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:15.901778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.901732 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:25:15.902860 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.902830 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:15.903457 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.903438 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:25:15.939960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.939033 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:15.939960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.939696 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:25:15.997139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:15.997078 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:25:16.025113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.025049 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.037086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.025776 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:25:16.037086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.027156 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:25:16.037086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.028192 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.044969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.043711 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:25:16.124113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.124060 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.396996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.392864 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:25:16.476486 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.476414 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:16.500959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.495124 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.500959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.479786 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:16.500959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.495520 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.643610 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.643494 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.704008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.687129 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:16.704008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.687601 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:25:16.894968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.891031 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.921316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.921273 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:16.922132 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.921203 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:25:16.939567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:16.939457 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:25:17.002855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.001095 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:25:17.030986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.027340 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.030986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.027800 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.030986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.030240 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.037452 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.036030 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.054817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.053801 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.130222 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.129533 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:25:17.323615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.320578 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:25:17.432975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.385260 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:25:17.432975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.396056 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.699969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.699622 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:25:17.941241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.941191 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:17.984322 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:17.983183 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:25:18.011103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.010167 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:25:18.056634 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.055561 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.056634 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.055961 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:25:18.078241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.066749 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.098609 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.098084 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.118957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.107182 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.382962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.382079 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.407962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.403622 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.434962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.434190 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.577498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.577454 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:18.577787 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.577754 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.663373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.663277 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:25:18.739188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.739138 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.969061 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:18.943036 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.015506 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.015462 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.100118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.100070 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:25:19.129959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.100702 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.155542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.112499 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:25:19.206004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.112550 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.206004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.112597 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:25:19.206004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.112616 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.206004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.146415 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.254159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.254039 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.432155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.432103 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.446231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.432430 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.744333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.744289 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:25:19.745862 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.745824 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.760077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.747013 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.779139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.779082 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:19.779487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.779436 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.871976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.862041 4883 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:19.871976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.862114 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.913640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.913586 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:19.944282 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:19.944233 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:20.051822 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.051720 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:20.052617 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.052585 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:25:20.053409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.053387 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:25:20.104080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.104029 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:25:20.234009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.210606 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:20.479244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.439653 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:25:20.479244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.441865 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:25:20.479244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.453363 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:25:20.500008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.499971 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:25:20.517914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.517881 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:25:20.521506 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.521447 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:20.523353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.523314 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:25:20.525868 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.525833 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:20.527257 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.527229 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:25:20.543054 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.543004 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 18:25:20.557658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.557627 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:25:20.559419 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.559394 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:25:20.559666 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.559650 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:25:20.561164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.561142 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:25:20.561359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.561344 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:25:20.561498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:20.561485 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:25:24.314660 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.314623 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg"] Oct 30 18:25:24.315242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.315227 4883 cpu_manager.go:414] "containerMap: removing container" podUID="b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77" containerName="revoke-keys" Oct 30 18:25:24.315287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.315243 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77" containerName="revoke-keys" Oct 30 18:25:24.315368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.315356 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77" containerName="revoke-keys" Oct 30 18:25:24.319260 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.319235 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.348171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.348051 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg"] Oct 30 18:25:24.351450 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.351421 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b8b654-openshift-default-dockercfg-lrm47\"" Oct 30 18:25:24.360422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.359025 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istiod-ca-cert\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.360422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.359081 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-envoy\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.360422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.359210 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-certs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.360422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.359261 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2qzt\" (UniqueName: \"kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-kube-api-access-g2qzt\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.360422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.359298 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.360422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.359340 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-data\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.360422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.359364 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-podinfo\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.360422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.359461 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-credential-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.360422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.359496 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-token\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.461575 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.461542 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-credential-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.461897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.461595 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-token\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.461897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.461717 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istiod-ca-cert\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.461897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.461747 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-envoy\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.461897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.461878 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-certs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.462493 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.461927 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2qzt\" (UniqueName: \"kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-kube-api-access-g2qzt\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.462493 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.461972 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.462493 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.462017 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-data\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.462493 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.462042 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-podinfo\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.462493 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.462093 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-credential-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.462493 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.462357 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-certs\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.463195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.463170 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-socket\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.463306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.463230 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-data\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.463535 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.463509 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istiod-ca-cert\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.465243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.465218 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-envoy\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.465340 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.465251 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-podinfo\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.488582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.488552 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-token\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.488762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.488551 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2qzt\" (UniqueName: \"kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-kube-api-access-g2qzt\") pod \"e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.633894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.633856 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:24.841860 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.841818 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c"] Oct 30 18:25:24.846705 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.846672 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.865397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.865362 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c2b103-openshift-default-dockercfg-2gsp2\"" Oct 30 18:25:24.866773 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.866741 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.867006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.866826 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.867006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.866895 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.867006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.866959 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.867721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.867689 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.867876 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.867848 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca77461-d958-4246-8ca8-0c45925a9e6c-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.867926 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.867903 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.868045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.868023 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.868118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.868102 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscz4\" (UniqueName: \"kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-kube-api-access-sscz4\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.870683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.870661 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c"] Oct 30 18:25:24.967904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.967868 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg"] Oct 30 18:25:24.968967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.968898 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969010 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969058 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca77461-d958-4246-8ca8-0c45925a9e6c-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969093 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969166 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969317 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:25:24.969262 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd998b018_884a_49d1_9e22_a6cc2a35ce6f.slice/crio-d71422c7f4165ce8ecaaf34e074a3e6eba607f3369a87125e062250a2693ce17 WatchSource:0}: Error finding container d71422c7f4165ce8ecaaf34e074a3e6eba607f3369a87125e062250a2693ce17: Status 404 returned error can't find the container with id d71422c7f4165ce8ecaaf34e074a3e6eba607f3369a87125e062250a2693ce17 Oct 30 18:25:24.969379 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969322 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969379 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969356 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969377 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscz4\" (UniqueName: \"kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-kube-api-access-sscz4\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969479 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969571 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969644 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969707 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969840 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969820 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca77461-d958-4246-8ca8-0c45925a9e6c-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.969887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.969854 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.972320 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.972299 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.972536 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.972519 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.972726 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.972692 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:24.972843 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.972806 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:24.972962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.972872 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:24.994288 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.994254 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscz4\" (UniqueName: \"kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-kube-api-access-sscz4\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:24.994430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:24.994310 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:25.163347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:25.163313 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:25.503421 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:25.503395 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c"] Oct 30 18:25:25.504858 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:25:25.504817 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca77461_d958_4246_8ca8_0c45925a9e6c.slice/crio-667750bc60a70ef847d131efdc42006de2637c177deeebc139438ebd7ff9dfe1 WatchSource:0}: Error finding container 667750bc60a70ef847d131efdc42006de2637c177deeebc139438ebd7ff9dfe1: Status 404 returned error can't find the container with id 667750bc60a70ef847d131efdc42006de2637c177deeebc139438ebd7ff9dfe1 Oct 30 18:25:25.507802 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:25.507766 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:25.507912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:25.507842 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:25.507912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:25.507870 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:25.586695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:25.586655 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" event={"ID":"7ca77461-d958-4246-8ca8-0c45925a9e6c","Type":"ContainerStarted","Data":"667750bc60a70ef847d131efdc42006de2637c177deeebc139438ebd7ff9dfe1"} Oct 30 18:25:25.588600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:25.588563 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" event={"ID":"d998b018-884a-49d1-9e22-a6cc2a35ce6f","Type":"ContainerStarted","Data":"66310b7d1a6fb816e4f1a083a21d20f8a26bab7f0414586fc1efd657eb3efa3a"} Oct 30 18:25:25.588757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:25.588637 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" event={"ID":"d998b018-884a-49d1-9e22-a6cc2a35ce6f","Type":"ContainerStarted","Data":"d71422c7f4165ce8ecaaf34e074a3e6eba607f3369a87125e062250a2693ce17"} Oct 30 18:25:25.636001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:25.634336 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:26.624964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:26.624896 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" event={"ID":"7ca77461-d958-4246-8ca8-0c45925a9e6c","Type":"ContainerStarted","Data":"b371823883b2993929be1ebe01afe19177e8ed9ca53ce7bbcbec75745036038c"} Oct 30 18:25:26.636055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:26.636019 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 18:25:26.636235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:26.636073 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" podUID="d998b018-884a-49d1-9e22-a6cc2a35ce6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 18:25:26.675901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:26.675828 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" podStartSLOduration=2.675803107 podStartE2EDuration="2.675803107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:25.640369846 +0000 UTC m=+1984.597373682" watchObservedRunningTime="2026-10-30 18:25:26.675803107 +0000 UTC m=+1985.632806944" Oct 30 18:25:26.684964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:26.682597 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" podStartSLOduration=2.682579817 podStartE2EDuration="2.682579817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:26.680965065 +0000 UTC m=+1985.637968883" watchObservedRunningTime="2026-10-30 18:25:26.682579817 +0000 UTC m=+1985.639583653" Oct 30 18:25:27.193959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:27.192899 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 18:25:27.193959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:27.192991 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" podUID="7ca77461-d958-4246-8ca8-0c45925a9e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 18:25:27.219747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:27.219716 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:27.634243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:27.634208 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 18:25:27.634716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:27.634688 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" podUID="d998b018-884a-49d1-9e22-a6cc2a35ce6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 18:25:28.164119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:28.164080 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 18:25:28.164418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:28.164388 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" podUID="7ca77461-d958-4246-8ca8-0c45925a9e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 18:25:28.635178 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:28.635139 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 18:25:28.635608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:28.635202 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" podUID="d998b018-884a-49d1-9e22-a6cc2a35ce6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 18:25:29.187959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:29.187396 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 18:25:29.187959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:29.187476 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" podUID="7ca77461-d958-4246-8ca8-0c45925a9e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 18:25:29.635603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:29.634779 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 18:25:29.635603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:29.634845 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" podUID="d998b018-884a-49d1-9e22-a6cc2a35ce6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 18:25:30.164028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:30.163986 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 18:25:30.164611 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:30.164285 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" podUID="7ca77461-d958-4246-8ca8-0c45925a9e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 18:25:30.634715 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:30.634672 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 18:25:30.635013 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:30.634990 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" podUID="d998b018-884a-49d1-9e22-a6cc2a35ce6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 18:25:31.170964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:31.163985 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 18:25:31.170964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:31.164053 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" podUID="7ca77461-d958-4246-8ca8-0c45925a9e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 18:25:31.718970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:31.718422 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:31.719864 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:31.719834 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:31.726090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:31.726017 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:32.321323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:32.320090 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:32.321757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:32.321508 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:32.323734 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:32.323705 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:25:32.505052 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:32.504989 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9"] Oct 30 18:25:32.509770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:32.509736 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b8b654-hkvn9"] Oct 30 18:25:33.180344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:33.180295 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6883fb30-8ec0-4491-8c1f-368997e160bd" path="/var/lib/kubelet/pods/6883fb30-8ec0-4491-8c1f-368997e160bd/volumes" Oct 30 18:25:34.991748 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:34.991711 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j"] Oct 30 18:25:34.997712 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:34.997684 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.015620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.015583 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7ffac8-openshift-default-dockercfg-8kwr2\"" Oct 30 18:25:35.036360 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.036320 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j"] Oct 30 18:25:35.090158 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:25:35.090116 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:25:35.112362 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.112326 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-podinfo\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.112523 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.112373 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-data\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.112639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.112516 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-certs\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.112639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.112549 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-token\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.112639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.112596 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfdt9\" (UniqueName: \"kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-kube-api-access-kfdt9\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.112742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.112726 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-credential-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.112778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.112764 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.112822 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.112809 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-envoy\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.112866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.112845 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istiod-ca-cert\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214037 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-podinfo\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214088 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-data\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214271 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-certs\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214352 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214313 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-token\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214403 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214387 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdt9\" (UniqueName: \"kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-kube-api-access-kfdt9\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214481 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-credential-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214518 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-data\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214531 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214576 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-envoy\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214590 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-certs\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214687 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istiod-ca-cert\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214822 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214764 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.214995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.214971 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-credential-socket\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.215372 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.215344 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istiod-ca-cert\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.216829 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.216801 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-podinfo\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.216988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.216830 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-envoy\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.238851 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.238812 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-token\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.239020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.238946 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdt9\" (UniqueName: \"kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-kube-api-access-kfdt9\") pod \"e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.312696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.312611 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:35.642221 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.642188 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j"] Oct 30 18:25:35.642974 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:25:35.642946 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc347ec_b9c5_4d35_828d_fdacaa649a45.slice/crio-998fc585c26818b3ff4331a442e3342231432b56cd40ad0de0869224bcdd3d31 WatchSource:0}: Error finding container 998fc585c26818b3ff4331a442e3342231432b56cd40ad0de0869224bcdd3d31: Status 404 returned error can't find the container with id 998fc585c26818b3ff4331a442e3342231432b56cd40ad0de0869224bcdd3d31 Oct 30 18:25:35.645698 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.645662 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:35.645791 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.645731 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:35.645791 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.645765 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:35.725007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:35.724973 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" event={"ID":"2cc347ec-b9c5-4d35-828d-fdacaa649a45","Type":"ContainerStarted","Data":"998fc585c26818b3ff4331a442e3342231432b56cd40ad0de0869224bcdd3d31"} Oct 30 18:25:36.733875 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:36.733836 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" event={"ID":"2cc347ec-b9c5-4d35-828d-fdacaa649a45","Type":"ContainerStarted","Data":"9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c"} Oct 30 18:25:36.807706 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:36.807646 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" podStartSLOduration=2.807631324 podStartE2EDuration="2.807631324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:36.805362371 +0000 UTC m=+1995.762366208" watchObservedRunningTime="2026-10-30 18:25:36.807631324 +0000 UTC m=+1995.764635160" Oct 30 18:25:37.312745 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:37.312705 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:37.314560 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:37.314523 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 18:25:37.314709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:37.314603 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" podUID="2cc347ec-b9c5-4d35-828d-fdacaa649a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 18:25:38.313632 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:38.313596 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 18:25:38.314159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:38.313649 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" podUID="2cc347ec-b9c5-4d35-828d-fdacaa649a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 18:25:39.313271 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:39.313232 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 18:25:39.313458 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:39.313300 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" podUID="2cc347ec-b9c5-4d35-828d-fdacaa649a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 18:25:39.933140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:39.933101 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7"] Oct 30 18:25:39.937733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:39.937709 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.030962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.028914 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7"] Oct 30 18:25:40.034908 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.034829 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.035211 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.035183 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6nq5\" (UniqueName: \"kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-kube-api-access-d6nq5\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.035345 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.035326 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.035628 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.035610 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.035795 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.035778 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.037588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.035976 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.037588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.036010 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.037588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.036415 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.037588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.036446 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36fb3684-6cf3-4d2d-b930-ad0388420357-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.063160 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.053823 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg"] Oct 30 18:25:40.063160 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.054117 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" podUID="d998b018-884a-49d1-9e22-a6cc2a35ce6f" containerName="istio-proxy" containerID="cri-o://66310b7d1a6fb816e4f1a083a21d20f8a26bab7f0414586fc1efd657eb3efa3a" gracePeriod=30 Oct 30 18:25:40.335497 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.335251 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.335984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.335803 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.336189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.336171 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.361368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.338793 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 18:25:40.361368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.339416 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" podUID="2cc347ec-b9c5-4d35-828d-fdacaa649a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 18:25:40.361368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.348399 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.361368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.348741 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.361368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.349059 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.361368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.349115 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36fb3684-6cf3-4d2d-b930-ad0388420357-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.361368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.349604 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.361368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.349971 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6nq5\" (UniqueName: \"kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-kube-api-access-d6nq5\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.361368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.354173 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-credential-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.361368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.354894 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-certs\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.361368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.359367 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-podinfo\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.361368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.360778 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-data\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.364616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.363577 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36fb3684-6cf3-4d2d-b930-ad0388420357-istiod-ca-cert\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.364616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.364144 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-socket\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.387989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.385120 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-token\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.428190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.428116 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-envoy\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.428383 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.428247 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6nq5\" (UniqueName: \"kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-kube-api-access-d6nq5\") pod \"e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.558350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.558310 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:40.707788 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.707742 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw"] Oct 30 18:25:40.714833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.714334 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.723052 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.723021 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw"] Oct 30 18:25:40.812536 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.812481 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.812728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.812610 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.812728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.812704 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.812808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.812731 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.812808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.812766 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6lzd\" (UniqueName: \"kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-kube-api-access-r6lzd\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.812914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.812885 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.812914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.812903 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.813005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.812922 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.813005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.812966 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.821561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.821503 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-77db82-openshift-default-dockercfg-btt49\"" Oct 30 18:25:40.915448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.914976 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.915448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.915069 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.915448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.915125 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6lzd\" (UniqueName: \"kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-kube-api-access-r6lzd\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.915448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.915278 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.915448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.915305 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.915448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.915332 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.915448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.915360 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.915448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.915397 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.915448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.915460 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.916820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.916786 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7"] Oct 30 18:25:40.917283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.917257 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.918535 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.918505 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.918919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.918897 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.919070 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.919047 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.919176 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.919136 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.924802 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.924772 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.925355 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:25:40.925331 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fb3684_6cf3_4d2d_b930_ad0388420357.slice/crio-fdad4cbdd84473a8c40fcd968429828340769c481cb4798c6e601cdfadaa418b WatchSource:0}: Error finding container fdad4cbdd84473a8c40fcd968429828340769c481cb4798c6e601cdfadaa418b: Status 404 returned error can't find the container with id fdad4cbdd84473a8c40fcd968429828340769c481cb4798c6e601cdfadaa418b Oct 30 18:25:40.928277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.928239 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:40.928439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.928317 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:40.928439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.928347 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:40.932060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.932032 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.945981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.942181 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:40.945981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:40.943363 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6lzd\" (UniqueName: \"kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-kube-api-access-r6lzd\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:41.044044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:41.041377 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:41.315010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:41.314965 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 18:25:41.315228 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:41.315048 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" podUID="2cc347ec-b9c5-4d35-828d-fdacaa649a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 18:25:41.548959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:41.542197 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:41.548959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:41.542312 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:41.548959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:41.542360 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:41.584317 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:41.574501 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw"] Oct 30 18:25:41.858987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:41.855205 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" event={"ID":"36fb3684-6cf3-4d2d-b930-ad0388420357","Type":"ContainerStarted","Data":"5dff6df0f2c7eac44e32d21ea545d3ea352421c1a597907131a95cae25324f5d"} Oct 30 18:25:41.858987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:41.855278 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" event={"ID":"36fb3684-6cf3-4d2d-b930-ad0388420357","Type":"ContainerStarted","Data":"fdad4cbdd84473a8c40fcd968429828340769c481cb4798c6e601cdfadaa418b"} Oct 30 18:25:41.899472 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:41.898584 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" event={"ID":"d73d31fd-a3de-4549-8d6d-bcddd36d3a48","Type":"ContainerStarted","Data":"da7ff5d34644e1314707a8b817cafc4045aa7b50167c03934041c8e1b9172d04"} Oct 30 18:25:41.950967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:41.944154 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" podStartSLOduration=2.944130481 podStartE2EDuration="2.944130481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:41.914098664 +0000 UTC m=+2000.871102503" watchObservedRunningTime="2026-10-30 18:25:41.944130481 +0000 UTC m=+2000.901134318" Oct 30 18:25:42.325982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:42.323073 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 18:25:42.325982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:42.323140 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" podUID="2cc347ec-b9c5-4d35-828d-fdacaa649a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 18:25:42.517057 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:42.517012 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c"] Oct 30 18:25:42.526844 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:42.523013 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7ffac8-ffm9c"] Oct 30 18:25:42.563038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:42.560383 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:42.563038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:42.562341 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:25:42.563038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:42.562398 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" podUID="36fb3684-6cf3-4d2d-b930-ad0388420357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 18:25:42.920504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:42.920261 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" event={"ID":"d73d31fd-a3de-4549-8d6d-bcddd36d3a48","Type":"ContainerStarted","Data":"ae60207c70c62d74f097143a236f5e58f5fd2f2b11a0196ed3493eb3fa656e99"} Oct 30 18:25:42.979969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:42.979433 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" podStartSLOduration=2.979412328 podStartE2EDuration="2.979412328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:42.978109947 +0000 UTC m=+2001.935113785" watchObservedRunningTime="2026-10-30 18:25:42.979412328 +0000 UTC m=+2001.936416166" Oct 30 18:25:43.049782 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:43.049109 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:43.051766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:43.051470 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 18:25:43.051766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:43.051524 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" podUID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 18:25:43.225249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:43.225178 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77" path="/var/lib/kubelet/pods/b6b4f1fb-6f3c-47cc-bc0c-421bddc2bd77/volumes" Oct 30 18:25:43.323116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:43.322540 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:43.324367 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:43.324344 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:43.325550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:43.325529 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:43.587006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:43.580303 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:25:43.587006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:43.580374 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" podUID="36fb3684-6cf3-4d2d-b930-ad0388420357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 18:25:44.043447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:44.043407 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 18:25:44.043972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:44.043467 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" podUID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 18:25:44.558717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:44.558669 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:25:44.558983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:44.558738 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" podUID="36fb3684-6cf3-4d2d-b930-ad0388420357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 18:25:45.046978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:45.045140 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 18:25:45.046978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:45.045215 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" podUID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 18:25:45.559378 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:45.559340 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:25:45.559695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:45.559666 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" podUID="36fb3684-6cf3-4d2d-b930-ad0388420357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 18:25:46.058168 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.058125 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 18:25:46.058655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.058193 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" podUID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 18:25:46.114971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.089865 4883 generic.go:356] "Generic (PLEG): container finished" podID="d998b018-884a-49d1-9e22-a6cc2a35ce6f" containerID="66310b7d1a6fb816e4f1a083a21d20f8a26bab7f0414586fc1efd657eb3efa3a" exitCode=0 Oct 30 18:25:46.114971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.090122 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" event={"ID":"d998b018-884a-49d1-9e22-a6cc2a35ce6f","Type":"ContainerDied","Data":"66310b7d1a6fb816e4f1a083a21d20f8a26bab7f0414586fc1efd657eb3efa3a"} Oct 30 18:25:46.114971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.090211 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" event={"ID":"d998b018-884a-49d1-9e22-a6cc2a35ce6f","Type":"ContainerDied","Data":"d71422c7f4165ce8ecaaf34e074a3e6eba607f3369a87125e062250a2693ce17"} Oct 30 18:25:46.114971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.090235 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d71422c7f4165ce8ecaaf34e074a3e6eba607f3369a87125e062250a2693ce17" Oct 30 18:25:46.114971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.093669 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:46.246904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.246807 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istiod-ca-cert\") pod \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " Oct 30 18:25:46.247108 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.246914 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-certs\") pod \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " Oct 30 18:25:46.247108 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.246984 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-kube-api-access-g2qzt\" (UniqueName: \"kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-kube-api-access-g2qzt\") pod \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " Oct 30 18:25:46.247108 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.247059 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-envoy\") pod \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " Oct 30 18:25:46.247292 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.247148 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-socket\") pod \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " Oct 30 18:25:46.247292 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.247219 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-token\" (UniqueName: \"kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-token\") pod \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " Oct 30 18:25:46.247403 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.247360 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-podinfo\") pod \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " Oct 30 18:25:46.247709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.247677 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istiod-ca-cert" pod "d998b018-884a-49d1-9e22-a6cc2a35ce6f" (UID: "d998b018-884a-49d1-9e22-a6cc2a35ce6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:46.251733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.247953 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-credential-socket\") pod \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " Oct 30 18:25:46.251733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.248037 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-data\") pod \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\" (UID: \"d998b018-884a-49d1-9e22-a6cc2a35ce6f\") " Oct 30 18:25:46.251733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.248276 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-certs" pod "d998b018-884a-49d1-9e22-a6cc2a35ce6f" (UID: "d998b018-884a-49d1-9e22-a6cc2a35ce6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:46.251733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.248506 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-data" pod "d998b018-884a-49d1-9e22-a6cc2a35ce6f" (UID: "d998b018-884a-49d1-9e22-a6cc2a35ce6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:46.251733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.248776 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.251733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.248796 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.251733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.248813 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.251733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.249064 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-credential-socket" pod "d998b018-884a-49d1-9e22-a6cc2a35ce6f" (UID: "d998b018-884a-49d1-9e22-a6cc2a35ce6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:46.251733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.249692 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-socket" pod "d998b018-884a-49d1-9e22-a6cc2a35ce6f" (UID: "d998b018-884a-49d1-9e22-a6cc2a35ce6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:46.252350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.252072 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-envoy" pod "d998b018-884a-49d1-9e22-a6cc2a35ce6f" (UID: "d998b018-884a-49d1-9e22-a6cc2a35ce6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:46.258024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.257917 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-kube-api-access-g2qzt" pod "d998b018-884a-49d1-9e22-a6cc2a35ce6f" (UID: "d998b018-884a-49d1-9e22-a6cc2a35ce6f"). InnerVolumeSpecName "kube-api-access-g2qzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:46.335008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.319952 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-token" pod "d998b018-884a-49d1-9e22-a6cc2a35ce6f" (UID: "d998b018-884a-49d1-9e22-a6cc2a35ce6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:46.335008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.325074 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-podinfo" pod "d998b018-884a-49d1-9e22-a6cc2a35ce6f" (UID: "d998b018-884a-49d1-9e22-a6cc2a35ce6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:46.351346 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.351313 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.351346 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.351341 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.351346 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.351351 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.351585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.351361 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d998b018-884a-49d1-9e22-a6cc2a35ce6f-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.351585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.351369 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d998b018-884a-49d1-9e22-a6cc2a35ce6f-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.351585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.351379 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2qzt\" (UniqueName: \"kubernetes.io/projected/d998b018-884a-49d1-9e22-a6cc2a35ce6f-kube-api-access-g2qzt\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:46.558902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.558814 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:25:46.558902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:46.558870 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" podUID="36fb3684-6cf3-4d2d-b930-ad0388420357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 18:25:47.051469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:47.049254 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 18:25:47.051469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:47.049361 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" podUID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 18:25:47.147064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:47.146109 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg" Oct 30 18:25:47.237451 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:47.237415 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg"] Oct 30 18:25:47.243131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:47.243093 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8b654-openshift-default-68f8b94cb5-vjnrg"] Oct 30 18:25:47.567731 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:47.567692 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:25:47.568051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:47.568014 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" podUID="36fb3684-6cf3-4d2d-b930-ad0388420357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 18:25:48.060959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.043742 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 18:25:48.060959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.043845 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" podUID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 18:25:48.662960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.662219 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:48.662960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.662271 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w"] Oct 30 18:25:48.662960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.662862 4883 cpu_manager.go:414] "containerMap: removing container" podUID="d998b018-884a-49d1-9e22-a6cc2a35ce6f" containerName="istio-proxy" Oct 30 18:25:48.662960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.662876 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d998b018-884a-49d1-9e22-a6cc2a35ce6f" containerName="istio-proxy" Oct 30 18:25:48.663507 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.663021 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="d998b018-884a-49d1-9e22-a6cc2a35ce6f" containerName="istio-proxy" Oct 30 18:25:48.684083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.673490 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:48.684083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.673578 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:25:48.684083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.673681 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:48.697960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.694384 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:25:48.703068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.703035 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:25:48.703325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.703116 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:25:48.704346 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.704317 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:25:48.704694 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.704591 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w"] Oct 30 18:25:48.817618 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.817567 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w\" (UID: \"9db71944-6f08-407e-9ee8-bbb39cc0e190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:48.817826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.817623 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4n9\" (UniqueName: \"kubernetes.io/projected/9db71944-6f08-407e-9ee8-bbb39cc0e190-kube-api-access-kt4n9\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w\" (UID: \"9db71944-6f08-407e-9ee8-bbb39cc0e190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:48.818083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.818041 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w\" (UID: \"9db71944-6f08-407e-9ee8-bbb39cc0e190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:48.919364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.919266 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w\" (UID: \"9db71944-6f08-407e-9ee8-bbb39cc0e190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:48.919546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.919424 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w\" (UID: \"9db71944-6f08-407e-9ee8-bbb39cc0e190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:48.919546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.919448 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4n9\" (UniqueName: \"kubernetes.io/projected/9db71944-6f08-407e-9ee8-bbb39cc0e190-kube-api-access-kt4n9\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w\" (UID: \"9db71944-6f08-407e-9ee8-bbb39cc0e190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:48.920207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.920177 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w\" (UID: \"9db71944-6f08-407e-9ee8-bbb39cc0e190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:48.922126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.922098 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w\" (UID: \"9db71944-6f08-407e-9ee8-bbb39cc0e190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:48.948068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:48.947996 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4n9\" (UniqueName: \"kubernetes.io/projected/9db71944-6f08-407e-9ee8-bbb39cc0e190-kube-api-access-kt4n9\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w\" (UID: \"9db71944-6f08-407e-9ee8-bbb39cc0e190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:49.024572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.024538 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:49.181164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.181074 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d998b018-884a-49d1-9e22-a6cc2a35ce6f" path="/var/lib/kubelet/pods/d998b018-884a-49d1-9e22-a6cc2a35ce6f/volumes" Oct 30 18:25:49.345389 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.345357 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w"] Oct 30 18:25:49.346986 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:25:49.346918 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db71944_6f08_407e_9ee8_bbb39cc0e190.slice/crio-8bd117c9989d1ea5db370c17e90eab0e690b0cfe89e8aca9d824b706ff254c08 WatchSource:0}: Error finding container 8bd117c9989d1ea5db370c17e90eab0e690b0cfe89e8aca9d824b706ff254c08: Status 404 returned error can't find the container with id 8bd117c9989d1ea5db370c17e90eab0e690b0cfe89e8aca9d824b706ff254c08 Oct 30 18:25:49.373782 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.373730 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:49.374336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.374275 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:49.375614 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.375591 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:25:49.456360 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.456281 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6"] Oct 30 18:25:49.462328 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.462298 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:49.487548 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.487512 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6"] Oct 30 18:25:49.635757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.635720 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv29k\" (UniqueName: \"kubernetes.io/projected/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-kube-api-access-gv29k\") pod \"maas-api-revoke-sub-2343eca5-78kv6\" (UID: \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:49.635958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.635894 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2343eca5-78kv6\" (UID: \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:49.636033 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.636015 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2343eca5-78kv6\" (UID: \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:49.737447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.737364 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv29k\" (UniqueName: \"kubernetes.io/projected/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-kube-api-access-gv29k\") pod \"maas-api-revoke-sub-2343eca5-78kv6\" (UID: \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:49.737922 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.737507 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2343eca5-78kv6\" (UID: \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:49.737922 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.737575 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2343eca5-78kv6\" (UID: \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:49.738334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.738310 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2343eca5-78kv6\" (UID: \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:49.740040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.740022 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2343eca5-78kv6\" (UID: \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:49.762773 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.762739 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv29k\" (UniqueName: \"kubernetes.io/projected/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-kube-api-access-gv29k\") pod \"maas-api-revoke-sub-2343eca5-78kv6\" (UID: \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:49.779397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:49.779356 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:50.088015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:50.087984 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6"] Oct 30 18:25:50.091532 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:25:50.091487 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2fb32e_690f_41d3_a07c_074cdb76f0d6.slice/crio-7d7f011784f92e617092f87cb7e8354d814cb3866ebbc061b2d21cb2d1f89eb0 WatchSource:0}: Error finding container 7d7f011784f92e617092f87cb7e8354d814cb3866ebbc061b2d21cb2d1f89eb0: Status 404 returned error can't find the container with id 7d7f011784f92e617092f87cb7e8354d814cb3866ebbc061b2d21cb2d1f89eb0 Oct 30 18:25:50.184642 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:50.184584 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" event={"ID":"9db71944-6f08-407e-9ee8-bbb39cc0e190","Type":"ContainerStarted","Data":"5f75582f2bee6b804d2665a25d916e25cd09b5ee4f7d8429ab913f642ef8d41d"} Oct 30 18:25:50.184888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:50.184679 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" event={"ID":"9db71944-6f08-407e-9ee8-bbb39cc0e190","Type":"ContainerStarted","Data":"8bd117c9989d1ea5db370c17e90eab0e690b0cfe89e8aca9d824b706ff254c08"} Oct 30 18:25:50.186884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:50.186635 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" event={"ID":"6c2fb32e-690f-41d3-a07c-074cdb76f0d6","Type":"ContainerStarted","Data":"7d7f011784f92e617092f87cb7e8354d814cb3866ebbc061b2d21cb2d1f89eb0"} Oct 30 18:25:50.248667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:50.248614 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" podStartSLOduration=2.248598495 podStartE2EDuration="2.248598495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:50.246991887 +0000 UTC m=+2009.203995732" watchObservedRunningTime="2026-10-30 18:25:50.248598495 +0000 UTC m=+2009.205602330" Oct 30 18:25:50.484992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:50.484952 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j"] Oct 30 18:25:50.485251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:50.485220 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" podUID="2cc347ec-b9c5-4d35-828d-fdacaa649a45" containerName="istio-proxy" containerID="cri-o://9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c" gracePeriod=30 Oct 30 18:25:51.196580 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:51.196545 4883 generic.go:356] "Generic (PLEG): container finished" podID="9db71944-6f08-407e-9ee8-bbb39cc0e190" containerID="5f75582f2bee6b804d2665a25d916e25cd09b5ee4f7d8429ab913f642ef8d41d" exitCode=0 Oct 30 18:25:51.196988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:51.196622 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" event={"ID":"9db71944-6f08-407e-9ee8-bbb39cc0e190","Type":"ContainerDied","Data":"5f75582f2bee6b804d2665a25d916e25cd09b5ee4f7d8429ab913f642ef8d41d"} Oct 30 18:25:51.198538 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:51.198511 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" event={"ID":"6c2fb32e-690f-41d3-a07c-074cdb76f0d6","Type":"ContainerStarted","Data":"03f6ef8b2b5fbf7134e197ef5413aa5bd9f1376a11ae941522098f58adb06a7e"} Oct 30 18:25:52.207884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.207847 4883 generic.go:356] "Generic (PLEG): container finished" podID="6c2fb32e-690f-41d3-a07c-074cdb76f0d6" containerID="03f6ef8b2b5fbf7134e197ef5413aa5bd9f1376a11ae941522098f58adb06a7e" exitCode=0 Oct 30 18:25:52.208428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.207952 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" event={"ID":"6c2fb32e-690f-41d3-a07c-074cdb76f0d6","Type":"ContainerDied","Data":"03f6ef8b2b5fbf7134e197ef5413aa5bd9f1376a11ae941522098f58adb06a7e"} Oct 30 18:25:52.444631 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.444590 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:52.498911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.498871 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:52.555957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.539449 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-kube-api-access-gv29k\" (UniqueName: \"kubernetes.io/projected/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-kube-api-access-gv29k\") pod \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\" (UID: \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\") " Oct 30 18:25:52.555957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.539594 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-openshift-ca\") pod \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\" (UID: \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\") " Oct 30 18:25:52.555957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.539665 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-xks-ca\") pod \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\" (UID: \"6c2fb32e-690f-41d3-a07c-074cdb76f0d6\") " Oct 30 18:25:52.555957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.539706 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db71944-6f08-407e-9ee8-bbb39cc0e190-kube-api-access-kt4n9\" (UniqueName: \"kubernetes.io/projected/9db71944-6f08-407e-9ee8-bbb39cc0e190-kube-api-access-kt4n9\") pod \"9db71944-6f08-407e-9ee8-bbb39cc0e190\" (UID: \"9db71944-6f08-407e-9ee8-bbb39cc0e190\") " Oct 30 18:25:52.555957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.539750 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-openshift-ca\") pod \"9db71944-6f08-407e-9ee8-bbb39cc0e190\" (UID: \"9db71944-6f08-407e-9ee8-bbb39cc0e190\") " Oct 30 18:25:52.555957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.539779 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-xks-ca\") pod \"9db71944-6f08-407e-9ee8-bbb39cc0e190\" (UID: \"9db71944-6f08-407e-9ee8-bbb39cc0e190\") " Oct 30 18:25:52.583187 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.582729 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-xks-ca" pod "9db71944-6f08-407e-9ee8-bbb39cc0e190" (UID: "9db71944-6f08-407e-9ee8-bbb39cc0e190"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:52.585606 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.585539 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-openshift-ca" pod "9db71944-6f08-407e-9ee8-bbb39cc0e190" (UID: "9db71944-6f08-407e-9ee8-bbb39cc0e190"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:52.586104 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.585929 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-openshift-ca" pod "6c2fb32e-690f-41d3-a07c-074cdb76f0d6" (UID: "6c2fb32e-690f-41d3-a07c-074cdb76f0d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:52.597254 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.597207 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-xks-ca" pod "6c2fb32e-690f-41d3-a07c-074cdb76f0d6" (UID: "6c2fb32e-690f-41d3-a07c-074cdb76f0d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:52.597254 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.597222 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-kube-api-access-gv29k" pod "6c2fb32e-690f-41d3-a07c-074cdb76f0d6" (UID: "6c2fb32e-690f-41d3-a07c-074cdb76f0d6"). InnerVolumeSpecName "kube-api-access-gv29k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:52.601194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.601160 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db71944-6f08-407e-9ee8-bbb39cc0e190-kube-api-access-kt4n9" pod "9db71944-6f08-407e-9ee8-bbb39cc0e190" (UID: "9db71944-6f08-407e-9ee8-bbb39cc0e190"). InnerVolumeSpecName "kube-api-access-kt4n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:52.642379 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.642250 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv29k\" (UniqueName: \"kubernetes.io/projected/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-kube-api-access-gv29k\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:52.642379 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.642292 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:52.642379 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.642308 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2fb32e-690f-41d3-a07c-074cdb76f0d6-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:52.642379 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.642323 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt4n9\" (UniqueName: \"kubernetes.io/projected/9db71944-6f08-407e-9ee8-bbb39cc0e190-kube-api-access-kt4n9\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:52.642379 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.642336 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:52.642379 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:52.642349 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db71944-6f08-407e-9ee8-bbb39cc0e190-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:53.219587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:53.219557 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" Oct 30 18:25:53.220056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:53.219592 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6" event={"ID":"6c2fb32e-690f-41d3-a07c-074cdb76f0d6","Type":"ContainerDied","Data":"7d7f011784f92e617092f87cb7e8354d814cb3866ebbc061b2d21cb2d1f89eb0"} Oct 30 18:25:53.220056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:53.219647 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d7f011784f92e617092f87cb7e8354d814cb3866ebbc061b2d21cb2d1f89eb0" Oct 30 18:25:53.221173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:53.221155 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" Oct 30 18:25:53.221281 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:53.221195 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w" event={"ID":"9db71944-6f08-407e-9ee8-bbb39cc0e190","Type":"ContainerDied","Data":"8bd117c9989d1ea5db370c17e90eab0e690b0cfe89e8aca9d824b706ff254c08"} Oct 30 18:25:53.221281 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:53.221237 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bd117c9989d1ea5db370c17e90eab0e690b0cfe89e8aca9d824b706ff254c08" Oct 30 18:25:56.199192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.199159 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:56.246021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.245981 4883 generic.go:356] "Generic (PLEG): container finished" podID="2cc347ec-b9c5-4d35-828d-fdacaa649a45" containerID="9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c" exitCode=0 Oct 30 18:25:56.246208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.246033 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" Oct 30 18:25:56.246208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.246081 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" event={"ID":"2cc347ec-b9c5-4d35-828d-fdacaa649a45","Type":"ContainerDied","Data":"9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c"} Oct 30 18:25:56.246208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.246141 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j" event={"ID":"2cc347ec-b9c5-4d35-828d-fdacaa649a45","Type":"ContainerDied","Data":"998fc585c26818b3ff4331a442e3342231432b56cd40ad0de0869224bcdd3d31"} Oct 30 18:25:56.246208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.246167 4883 scope.go:122] "RemoveContainer" containerID="9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c" Oct 30 18:25:56.257380 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.257358 4883 scope.go:122] "RemoveContainer" containerID="9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c" Oct 30 18:25:56.257755 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:25:56.257727 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c\": container with ID starting with 9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c not found: ID does not exist" containerID="9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c" Oct 30 18:25:56.257798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.257765 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c"} err="failed to get container status \"9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c\": rpc error: code = NotFound desc = could not find container \"9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c\": container with ID starting with 9af902284c895ba21a1433da4282a411456c9c41d06353e67371957e88f8ff0c not found: ID does not exist" Oct 30 18:25:56.306653 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.306548 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-kube-api-access-kfdt9\" (UniqueName: \"kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-kube-api-access-kfdt9\") pod \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " Oct 30 18:25:56.306653 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.306647 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-certs\") pod \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " Oct 30 18:25:56.306901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.306769 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istiod-ca-cert\") pod \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " Oct 30 18:25:56.306901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.306855 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-socket\") pod \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " Oct 30 18:25:56.307097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.306965 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-token\" (UniqueName: \"kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-token\") pod \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " Oct 30 18:25:56.307097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.307020 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-data\") pod \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " Oct 30 18:25:56.307097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.307044 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-certs" pod "2cc347ec-b9c5-4d35-828d-fdacaa649a45" (UID: "2cc347ec-b9c5-4d35-828d-fdacaa649a45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:56.307097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.307092 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-envoy\") pod \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " Oct 30 18:25:56.307385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.307130 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-podinfo\") pod \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " Oct 30 18:25:56.307385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.307197 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-credential-socket\") pod \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\" (UID: \"2cc347ec-b9c5-4d35-828d-fdacaa649a45\") " Oct 30 18:25:56.307385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.307232 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-socket" pod "2cc347ec-b9c5-4d35-828d-fdacaa649a45" (UID: "2cc347ec-b9c5-4d35-828d-fdacaa649a45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:56.307385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.307312 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istiod-ca-cert" pod "2cc347ec-b9c5-4d35-828d-fdacaa649a45" (UID: "2cc347ec-b9c5-4d35-828d-fdacaa649a45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:56.307591 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.307571 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-data" pod "2cc347ec-b9c5-4d35-828d-fdacaa649a45" (UID: "2cc347ec-b9c5-4d35-828d-fdacaa649a45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:56.308093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.308067 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:56.308224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.308099 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:56.308224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.308113 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:56.308224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.308128 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:56.308996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.308970 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-credential-socket" pod "2cc347ec-b9c5-4d35-828d-fdacaa649a45" (UID: "2cc347ec-b9c5-4d35-828d-fdacaa649a45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:56.310049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.310019 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-envoy" pod "2cc347ec-b9c5-4d35-828d-fdacaa649a45" (UID: "2cc347ec-b9c5-4d35-828d-fdacaa649a45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:56.310300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.310267 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-podinfo" pod "2cc347ec-b9c5-4d35-828d-fdacaa649a45" (UID: "2cc347ec-b9c5-4d35-828d-fdacaa649a45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:56.310451 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.310430 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-kube-api-access-kfdt9" pod "2cc347ec-b9c5-4d35-828d-fdacaa649a45" (UID: "2cc347ec-b9c5-4d35-828d-fdacaa649a45"). InnerVolumeSpecName "kube-api-access-kfdt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:56.313064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.313041 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-token" pod "2cc347ec-b9c5-4d35-828d-fdacaa649a45" (UID: "2cc347ec-b9c5-4d35-828d-fdacaa649a45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:56.410053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.410011 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfdt9\" (UniqueName: \"kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-kube-api-access-kfdt9\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:56.410053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.410054 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:56.410278 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.410067 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:56.410278 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.410079 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc347ec-b9c5-4d35-828d-fdacaa649a45-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:56.410278 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.410091 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc347ec-b9c5-4d35-828d-fdacaa649a45-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:25:56.600240 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.600201 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j"] Oct 30 18:25:56.615805 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:56.615765 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7ffac8-openshift-default-767586db6f-kp62j"] Oct 30 18:25:57.186231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.186191 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc347ec-b9c5-4d35-828d-fdacaa649a45" path="/var/lib/kubelet/pods/2cc347ec-b9c5-4d35-828d-fdacaa649a45/volumes" Oct 30 18:25:57.938077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.938044 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms"] Oct 30 18:25:57.938646 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.938631 4883 cpu_manager.go:414] "containerMap: removing container" podUID="6c2fb32e-690f-41d3-a07c-074cdb76f0d6" containerName="revoke-keys" Oct 30 18:25:57.938708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.938647 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2fb32e-690f-41d3-a07c-074cdb76f0d6" containerName="revoke-keys" Oct 30 18:25:57.938708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.938662 4883 cpu_manager.go:414] "containerMap: removing container" podUID="2cc347ec-b9c5-4d35-828d-fdacaa649a45" containerName="istio-proxy" Oct 30 18:25:57.938708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.938667 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc347ec-b9c5-4d35-828d-fdacaa649a45" containerName="istio-proxy" Oct 30 18:25:57.938708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.938676 4883 cpu_manager.go:414] "containerMap: removing container" podUID="9db71944-6f08-407e-9ee8-bbb39cc0e190" containerName="revoke-keys" Oct 30 18:25:57.938708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.938682 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db71944-6f08-407e-9ee8-bbb39cc0e190" containerName="revoke-keys" Oct 30 18:25:57.938854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.938782 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc347ec-b9c5-4d35-828d-fdacaa649a45" containerName="istio-proxy" Oct 30 18:25:57.938854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.938791 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db71944-6f08-407e-9ee8-bbb39cc0e190" containerName="revoke-keys" Oct 30 18:25:57.938854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.938806 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2fb32e-690f-41d3-a07c-074cdb76f0d6" containerName="revoke-keys" Oct 30 18:25:57.943856 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.943832 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:25:57.962733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.962695 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms"] Oct 30 18:25:57.971686 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.971653 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:25:57.971917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.971658 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:25:57.971917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.971732 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:25:57.972114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:57.971814 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:25:58.039014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.038980 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4rs5\" (UniqueName: \"kubernetes.io/projected/bc985ab4-6938-44a5-9a9c-89530885929e-kube-api-access-w4rs5\") pod \"maas-api-revoke-sub-63c852d1-hbfms\" (UID: \"bc985ab4-6938-44a5-9a9c-89530885929e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:25:58.039216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.039034 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63c852d1-hbfms\" (UID: \"bc985ab4-6938-44a5-9a9c-89530885929e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:25:58.039216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.039181 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63c852d1-hbfms\" (UID: \"bc985ab4-6938-44a5-9a9c-89530885929e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:25:58.140667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.140623 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63c852d1-hbfms\" (UID: \"bc985ab4-6938-44a5-9a9c-89530885929e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:25:58.140844 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.140828 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4rs5\" (UniqueName: \"kubernetes.io/projected/bc985ab4-6938-44a5-9a9c-89530885929e-kube-api-access-w4rs5\") pod \"maas-api-revoke-sub-63c852d1-hbfms\" (UID: \"bc985ab4-6938-44a5-9a9c-89530885929e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:25:58.140964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.140889 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63c852d1-hbfms\" (UID: \"bc985ab4-6938-44a5-9a9c-89530885929e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:25:58.141548 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.141523 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63c852d1-hbfms\" (UID: \"bc985ab4-6938-44a5-9a9c-89530885929e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:25:58.143261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.143240 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63c852d1-hbfms\" (UID: \"bc985ab4-6938-44a5-9a9c-89530885929e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:25:58.165726 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.165695 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4rs5\" (UniqueName: \"kubernetes.io/projected/bc985ab4-6938-44a5-9a9c-89530885929e-kube-api-access-w4rs5\") pod \"maas-api-revoke-sub-63c852d1-hbfms\" (UID: \"bc985ab4-6938-44a5-9a9c-89530885929e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:25:58.260703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.260619 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:25:58.468275 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.468238 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh"] Oct 30 18:25:58.468477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.468453 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerName="istio-proxy" containerID="cri-o://3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a" gracePeriod=30 Oct 30 18:25:58.573593 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:58.573062 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms"] Oct 30 18:25:58.592136 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:25:58.592098 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc985ab4_6938_44a5_9a9c_89530885929e.slice/crio-cb544adea2611c5344dc0e356613696379b4b7e0f2f4671c46108d5c2e867716 WatchSource:0}: Error finding container cb544adea2611c5344dc0e356613696379b4b7e0f2f4671c46108d5c2e867716: Status 404 returned error can't find the container with id cb544adea2611c5344dc0e356613696379b4b7e0f2f4671c46108d5c2e867716 Oct 30 18:25:59.272528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.272436 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" event={"ID":"bc985ab4-6938-44a5-9a9c-89530885929e","Type":"ContainerStarted","Data":"4f4f9a7090575afb33494fc62677bf8ec16bb924fd7ec55aeeb35f01bca61e69"} Oct 30 18:25:59.272528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.272508 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" event={"ID":"bc985ab4-6938-44a5-9a9c-89530885929e","Type":"ContainerStarted","Data":"cb544adea2611c5344dc0e356613696379b4b7e0f2f4671c46108d5c2e867716"} Oct 30 18:25:59.315515 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.315430 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" podStartSLOduration=2.315373654 podStartE2EDuration="2.315373654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:59.315021233 +0000 UTC m=+2018.272025071" watchObservedRunningTime="2026-10-30 18:25:59.315373654 +0000 UTC m=+2018.272377492" Oct 30 18:25:59.642823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.642774 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk"] Oct 30 18:25:59.647909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.647881 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:25:59.663418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.663357 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk"] Oct 30 18:25:59.778544 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.778505 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6jq5\" (UniqueName: \"kubernetes.io/projected/4947af53-5359-452c-a822-8a3fb827c393-kube-api-access-g6jq5\") pod \"maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk\" (UID: \"4947af53-5359-452c-a822-8a3fb827c393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:25:59.778743 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.778567 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4947af53-5359-452c-a822-8a3fb827c393-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk\" (UID: \"4947af53-5359-452c-a822-8a3fb827c393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:25:59.778743 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.778657 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4947af53-5359-452c-a822-8a3fb827c393-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk\" (UID: \"4947af53-5359-452c-a822-8a3fb827c393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:25:59.879824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.879783 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6jq5\" (UniqueName: \"kubernetes.io/projected/4947af53-5359-452c-a822-8a3fb827c393-kube-api-access-g6jq5\") pod \"maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk\" (UID: \"4947af53-5359-452c-a822-8a3fb827c393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:25:59.880037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.879850 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4947af53-5359-452c-a822-8a3fb827c393-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk\" (UID: \"4947af53-5359-452c-a822-8a3fb827c393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:25:59.880037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.879878 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4947af53-5359-452c-a822-8a3fb827c393-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk\" (UID: \"4947af53-5359-452c-a822-8a3fb827c393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:25:59.880625 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.880597 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4947af53-5359-452c-a822-8a3fb827c393-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk\" (UID: \"4947af53-5359-452c-a822-8a3fb827c393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:25:59.882307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.882283 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4947af53-5359-452c-a822-8a3fb827c393-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk\" (UID: \"4947af53-5359-452c-a822-8a3fb827c393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:25:59.904796 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.904722 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6jq5\" (UniqueName: \"kubernetes.io/projected/4947af53-5359-452c-a822-8a3fb827c393-kube-api-access-g6jq5\") pod \"maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk\" (UID: \"4947af53-5359-452c-a822-8a3fb827c393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:25:59.972958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:25:59.972491 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:26:00.287826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:00.287787 4883 generic.go:356] "Generic (PLEG): container finished" podID="bc985ab4-6938-44a5-9a9c-89530885929e" containerID="4f4f9a7090575afb33494fc62677bf8ec16bb924fd7ec55aeeb35f01bca61e69" exitCode=0 Oct 30 18:26:00.288279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:00.287864 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" event={"ID":"bc985ab4-6938-44a5-9a9c-89530885929e","Type":"ContainerDied","Data":"4f4f9a7090575afb33494fc62677bf8ec16bb924fd7ec55aeeb35f01bca61e69"} Oct 30 18:26:00.436772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:00.436734 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk"] Oct 30 18:26:00.438404 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:26:00.438369 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4947af53_5359_452c_a822_8a3fb827c393.slice/crio-c6493d237dde72dab665f0a25ec605b8cbfcbb3c7c638dbd09f8c6b92d485fb6 WatchSource:0}: Error finding container c6493d237dde72dab665f0a25ec605b8cbfcbb3c7c638dbd09f8c6b92d485fb6: Status 404 returned error can't find the container with id c6493d237dde72dab665f0a25ec605b8cbfcbb3c7c638dbd09f8c6b92d485fb6 Oct 30 18:26:01.298293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.298255 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" event={"ID":"4947af53-5359-452c-a822-8a3fb827c393","Type":"ContainerStarted","Data":"2396dd330476af89031750aca863b6a8689d1133d5dcc8a2c60e9b3b49a9008e"} Oct 30 18:26:01.298817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.298304 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" event={"ID":"4947af53-5359-452c-a822-8a3fb827c393","Type":"ContainerStarted","Data":"c6493d237dde72dab665f0a25ec605b8cbfcbb3c7c638dbd09f8c6b92d485fb6"} Oct 30 18:26:01.343884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.343828 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" podStartSLOduration=2.343813955 podStartE2EDuration="2.343813955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:01.341174427 +0000 UTC m=+2020.298178257" watchObservedRunningTime="2026-10-30 18:26:01.343813955 +0000 UTC m=+2020.300817915" Oct 30 18:26:01.491744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.491713 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:26:01.515627 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.515581 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-openshift-ca\") pod \"bc985ab4-6938-44a5-9a9c-89530885929e\" (UID: \"bc985ab4-6938-44a5-9a9c-89530885929e\") " Oct 30 18:26:01.515818 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.515680 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-xks-ca\") pod \"bc985ab4-6938-44a5-9a9c-89530885929e\" (UID: \"bc985ab4-6938-44a5-9a9c-89530885929e\") " Oct 30 18:26:01.515874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.515858 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc985ab4-6938-44a5-9a9c-89530885929e-kube-api-access-w4rs5\" (UniqueName: \"kubernetes.io/projected/bc985ab4-6938-44a5-9a9c-89530885929e-kube-api-access-w4rs5\") pod \"bc985ab4-6938-44a5-9a9c-89530885929e\" (UID: \"bc985ab4-6938-44a5-9a9c-89530885929e\") " Oct 30 18:26:01.516069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.516035 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-openshift-ca" pod "bc985ab4-6938-44a5-9a9c-89530885929e" (UID: "bc985ab4-6938-44a5-9a9c-89530885929e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:01.516508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.516491 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.518494 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.518456 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-xks-ca" pod "bc985ab4-6938-44a5-9a9c-89530885929e" (UID: "bc985ab4-6938-44a5-9a9c-89530885929e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:01.518494 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.518490 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc985ab4-6938-44a5-9a9c-89530885929e-kube-api-access-w4rs5" pod "bc985ab4-6938-44a5-9a9c-89530885929e" (UID: "bc985ab4-6938-44a5-9a9c-89530885929e"). InnerVolumeSpecName "kube-api-access-w4rs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:01.618006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.617922 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc985ab4-6938-44a5-9a9c-89530885929e-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.618006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:01.618001 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4rs5\" (UniqueName: \"kubernetes.io/projected/bc985ab4-6938-44a5-9a9c-89530885929e-kube-api-access-w4rs5\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.307191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:02.307160 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" Oct 30 18:26:02.307667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:02.307161 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms" event={"ID":"bc985ab4-6938-44a5-9a9c-89530885929e","Type":"ContainerDied","Data":"cb544adea2611c5344dc0e356613696379b4b7e0f2f4671c46108d5c2e867716"} Oct 30 18:26:02.307667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:02.307304 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb544adea2611c5344dc0e356613696379b4b7e0f2f4671c46108d5c2e867716" Oct 30 18:26:02.308981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:02.308958 4883 generic.go:356] "Generic (PLEG): container finished" podID="4947af53-5359-452c-a822-8a3fb827c393" containerID="2396dd330476af89031750aca863b6a8689d1133d5dcc8a2c60e9b3b49a9008e" exitCode=0 Oct 30 18:26:02.309083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:02.309041 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" event={"ID":"4947af53-5359-452c-a822-8a3fb827c393","Type":"ContainerDied","Data":"2396dd330476af89031750aca863b6a8689d1133d5dcc8a2c60e9b3b49a9008e"} Oct 30 18:26:02.953961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:02.953207 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cntbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:26:02.953961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:02.953335 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:26:03.566666 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.566638 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:26:03.674770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.673585 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4947af53-5359-452c-a822-8a3fb827c393-kube-api-access-g6jq5\" (UniqueName: \"kubernetes.io/projected/4947af53-5359-452c-a822-8a3fb827c393-kube-api-access-g6jq5\") pod \"4947af53-5359-452c-a822-8a3fb827c393\" (UID: \"4947af53-5359-452c-a822-8a3fb827c393\") " Oct 30 18:26:03.674770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.673894 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4947af53-5359-452c-a822-8a3fb827c393-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4947af53-5359-452c-a822-8a3fb827c393-maas-api-xks-ca\") pod \"4947af53-5359-452c-a822-8a3fb827c393\" (UID: \"4947af53-5359-452c-a822-8a3fb827c393\") " Oct 30 18:26:03.674770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.673960 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4947af53-5359-452c-a822-8a3fb827c393-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4947af53-5359-452c-a822-8a3fb827c393-maas-api-openshift-ca\") pod \"4947af53-5359-452c-a822-8a3fb827c393\" (UID: \"4947af53-5359-452c-a822-8a3fb827c393\") " Oct 30 18:26:03.675277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.675222 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4947af53-5359-452c-a822-8a3fb827c393-maas-api-openshift-ca" pod "4947af53-5359-452c-a822-8a3fb827c393" (UID: "4947af53-5359-452c-a822-8a3fb827c393"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:03.677878 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.677849 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4947af53-5359-452c-a822-8a3fb827c393-kube-api-access-g6jq5" pod "4947af53-5359-452c-a822-8a3fb827c393" (UID: "4947af53-5359-452c-a822-8a3fb827c393"). InnerVolumeSpecName "kube-api-access-g6jq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:03.678048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.677886 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4947af53-5359-452c-a822-8a3fb827c393-maas-api-xks-ca" pod "4947af53-5359-452c-a822-8a3fb827c393" (UID: "4947af53-5359-452c-a822-8a3fb827c393"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:03.777959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.777204 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6jq5\" (UniqueName: \"kubernetes.io/projected/4947af53-5359-452c-a822-8a3fb827c393-kube-api-access-g6jq5\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.777959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.777249 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4947af53-5359-452c-a822-8a3fb827c393-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.777959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.777264 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4947af53-5359-452c-a822-8a3fb827c393-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.928637 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.928609 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:26:03.980265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.980217 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-data\") pod \"a5beef11-370d-4e74-8d12-e0541cb2a653\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " Oct 30 18:26:03.980456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.980291 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5beef11-370d-4e74-8d12-e0541cb2a653-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5beef11-370d-4e74-8d12-e0541cb2a653-istiod-ca-cert\") pod \"a5beef11-370d-4e74-8d12-e0541cb2a653\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " Oct 30 18:26:03.980517 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.980500 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-data" pod "a5beef11-370d-4e74-8d12-e0541cb2a653" (UID: "a5beef11-370d-4e74-8d12-e0541cb2a653"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:03.980639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.980620 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-kube-api-access-bhrfz\" (UniqueName: \"kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-kube-api-access-bhrfz\") pod \"a5beef11-370d-4e74-8d12-e0541cb2a653\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " Oct 30 18:26:03.980726 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.980708 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-token\" (UniqueName: \"kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-token\") pod \"a5beef11-370d-4e74-8d12-e0541cb2a653\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " Oct 30 18:26:03.980788 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.980736 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5beef11-370d-4e74-8d12-e0541cb2a653-istiod-ca-cert" pod "a5beef11-370d-4e74-8d12-e0541cb2a653" (UID: "a5beef11-370d-4e74-8d12-e0541cb2a653"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:03.980788 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.980745 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-podinfo\") pod \"a5beef11-370d-4e74-8d12-e0541cb2a653\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " Oct 30 18:26:03.980885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.980818 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-credential-socket\") pod \"a5beef11-370d-4e74-8d12-e0541cb2a653\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " Oct 30 18:26:03.980885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.980854 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-socket\") pod \"a5beef11-370d-4e74-8d12-e0541cb2a653\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " Oct 30 18:26:03.981008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.980889 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-certs\") pod \"a5beef11-370d-4e74-8d12-e0541cb2a653\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " Oct 30 18:26:03.981008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.980969 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-envoy\") pod \"a5beef11-370d-4e74-8d12-e0541cb2a653\" (UID: \"a5beef11-370d-4e74-8d12-e0541cb2a653\") " Oct 30 18:26:03.981314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.981284 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-socket" pod "a5beef11-370d-4e74-8d12-e0541cb2a653" (UID: "a5beef11-370d-4e74-8d12-e0541cb2a653"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:03.981528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.981503 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-credential-socket" pod "a5beef11-370d-4e74-8d12-e0541cb2a653" (UID: "a5beef11-370d-4e74-8d12-e0541cb2a653"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:03.981528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.981516 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-certs" pod "a5beef11-370d-4e74-8d12-e0541cb2a653" (UID: "a5beef11-370d-4e74-8d12-e0541cb2a653"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:03.981748 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.981724 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.981810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.981750 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.981810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.981766 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.981810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.981778 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.981810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.981788 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5beef11-370d-4e74-8d12-e0541cb2a653-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.984005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.983923 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-token" pod "a5beef11-370d-4e74-8d12-e0541cb2a653" (UID: "a5beef11-370d-4e74-8d12-e0541cb2a653"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:03.984606 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.984577 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-envoy" pod "a5beef11-370d-4e74-8d12-e0541cb2a653" (UID: "a5beef11-370d-4e74-8d12-e0541cb2a653"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:03.984773 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.984742 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-kube-api-access-bhrfz" pod "a5beef11-370d-4e74-8d12-e0541cb2a653" (UID: "a5beef11-370d-4e74-8d12-e0541cb2a653"). InnerVolumeSpecName "kube-api-access-bhrfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:03.985593 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:03.985562 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-podinfo" pod "a5beef11-370d-4e74-8d12-e0541cb2a653" (UID: "a5beef11-370d-4e74-8d12-e0541cb2a653"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:04.083310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.083270 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhrfz\" (UniqueName: \"kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-kube-api-access-bhrfz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.083310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.083307 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.083310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.083319 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.083566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.083327 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5beef11-370d-4e74-8d12-e0541cb2a653-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.326329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.326246 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" Oct 30 18:26:04.326329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.326270 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk" event={"ID":"4947af53-5359-452c-a822-8a3fb827c393","Type":"ContainerDied","Data":"c6493d237dde72dab665f0a25ec605b8cbfcbb3c7c638dbd09f8c6b92d485fb6"} Oct 30 18:26:04.326329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.326323 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6493d237dde72dab665f0a25ec605b8cbfcbb3c7c638dbd09f8c6b92d485fb6" Oct 30 18:26:04.327978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.327928 4883 generic.go:356] "Generic (PLEG): container finished" podID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerID="3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a" exitCode=0 Oct 30 18:26:04.328087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.327992 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" Oct 30 18:26:04.328125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.327991 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" event={"ID":"a5beef11-370d-4e74-8d12-e0541cb2a653","Type":"ContainerDied","Data":"3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a"} Oct 30 18:26:04.328159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.328139 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh" event={"ID":"a5beef11-370d-4e74-8d12-e0541cb2a653","Type":"ContainerDied","Data":"4cb4362ed910171eda91b09cd61dcdc5fa6b141fa492a2771ae10aaede8327cf"} Oct 30 18:26:04.328198 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.328180 4883 scope.go:122] "RemoveContainer" containerID="3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a" Oct 30 18:26:04.340097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.340074 4883 scope.go:122] "RemoveContainer" containerID="3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a" Oct 30 18:26:04.340390 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:26:04.340359 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a\": container with ID starting with 3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a not found: ID does not exist" containerID="3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a" Oct 30 18:26:04.340475 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.340401 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a"} err="failed to get container status \"3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a\": rpc error: code = NotFound desc = could not find container \"3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a\": container with ID starting with 3b703f4dd4e86c019d994951fe8b7ec210c0cabaab3909cc8838a64f6c553d7a not found: ID does not exist" Oct 30 18:26:04.385959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.385901 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh"] Oct 30 18:26:04.399783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:04.399746 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cntbh"] Oct 30 18:26:05.081231 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:26:05.081198 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:26:05.178819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:05.178785 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" path="/var/lib/kubelet/pods/a5beef11-370d-4e74-8d12-e0541cb2a653/volumes" Oct 30 18:26:12.910340 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.910303 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz"] Oct 30 18:26:12.910972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.910925 4883 cpu_manager.go:414] "containerMap: removing container" podUID="bc985ab4-6938-44a5-9a9c-89530885929e" containerName="revoke-keys" Oct 30 18:26:12.910972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.910955 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc985ab4-6938-44a5-9a9c-89530885929e" containerName="revoke-keys" Oct 30 18:26:12.911077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.910973 4883 cpu_manager.go:414] "containerMap: removing container" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerName="istio-proxy" Oct 30 18:26:12.911077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.910981 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerName="istio-proxy" Oct 30 18:26:12.911077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.911012 4883 cpu_manager.go:414] "containerMap: removing container" podUID="4947af53-5359-452c-a822-8a3fb827c393" containerName="revoke-keys" Oct 30 18:26:12.911077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.911017 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4947af53-5359-452c-a822-8a3fb827c393" containerName="revoke-keys" Oct 30 18:26:12.911265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.911125 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="4947af53-5359-452c-a822-8a3fb827c393" containerName="revoke-keys" Oct 30 18:26:12.911265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.911136 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc985ab4-6938-44a5-9a9c-89530885929e" containerName="revoke-keys" Oct 30 18:26:12.911265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.911144 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5beef11-370d-4e74-8d12-e0541cb2a653" containerName="istio-proxy" Oct 30 18:26:12.916337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.916316 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:12.935329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.935288 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz"] Oct 30 18:26:12.944283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.944251 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:26:12.944512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.944250 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:26:12.944612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.944409 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:26:12.945946 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.945911 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:26:12.947504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.947471 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wv9\" (UniqueName: \"kubernetes.io/projected/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-kube-api-access-66wv9\") pod \"maas-api-revoke-sub-5fb7d75f-pg2dz\" (UID: \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:12.947593 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.947579 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb7d75f-pg2dz\" (UID: \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:12.947767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.947749 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb7d75f-pg2dz\" (UID: \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:12.982422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.982383 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk"] Oct 30 18:26:12.987728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:12.987694 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1fae38-j85jk"] Oct 30 18:26:13.048911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:13.048865 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb7d75f-pg2dz\" (UID: \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:13.049113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:13.049020 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wv9\" (UniqueName: \"kubernetes.io/projected/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-kube-api-access-66wv9\") pod \"maas-api-revoke-sub-5fb7d75f-pg2dz\" (UID: \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:13.049170 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:13.049130 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb7d75f-pg2dz\" (UID: \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:13.049640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:13.049612 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb7d75f-pg2dz\" (UID: \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:13.051586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:13.051560 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb7d75f-pg2dz\" (UID: \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:13.103009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:13.095897 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wv9\" (UniqueName: \"kubernetes.io/projected/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-kube-api-access-66wv9\") pod \"maas-api-revoke-sub-5fb7d75f-pg2dz\" (UID: \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:13.179538 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:13.179444 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4947af53-5359-452c-a822-8a3fb827c393" path="/var/lib/kubelet/pods/4947af53-5359-452c-a822-8a3fb827c393/volumes" Oct 30 18:26:13.230255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:13.230221 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:13.547627 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:13.547587 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz"] Oct 30 18:26:13.549175 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:26:13.549145 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20aed91f_cb4d_4f9d_bb1a_fc8ca3b0aad0.slice/crio-ebd16f8d4b8c98147806586951799e321c162cb06e7e8089ff6a8e5183951624 WatchSource:0}: Error finding container ebd16f8d4b8c98147806586951799e321c162cb06e7e8089ff6a8e5183951624: Status 404 returned error can't find the container with id ebd16f8d4b8c98147806586951799e321c162cb06e7e8089ff6a8e5183951624 Oct 30 18:26:14.476539 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:14.476503 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" event={"ID":"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0","Type":"ContainerStarted","Data":"950127894fe22f35fa3f6b81983def9c19b0ef6b3ae85354782ecd144b1946e3"} Oct 30 18:26:14.477083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:14.476572 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" event={"ID":"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0","Type":"ContainerStarted","Data":"ebd16f8d4b8c98147806586951799e321c162cb06e7e8089ff6a8e5183951624"} Oct 30 18:26:14.520854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:14.520784 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" podStartSLOduration=2.520767335 podStartE2EDuration="2.520767335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:14.519575167 +0000 UTC m=+2033.476579003" watchObservedRunningTime="2026-10-30 18:26:14.520767335 +0000 UTC m=+2033.477771170" Oct 30 18:26:14.907514 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:14.907470 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp"] Oct 30 18:26:14.913419 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:14.912985 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:14.930620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:14.930589 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp"] Oct 30 18:26:14.988964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:14.988910 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b410d4a0-7tsxp\" (UID: \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:14.989171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:14.989037 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jf59\" (UniqueName: \"kubernetes.io/projected/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-kube-api-access-9jf59\") pod \"maas-api-revoke-sub-b410d4a0-7tsxp\" (UID: \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:14.989171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:14.989110 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b410d4a0-7tsxp\" (UID: \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:15.090273 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:15.090236 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b410d4a0-7tsxp\" (UID: \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:15.090273 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:15.090287 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jf59\" (UniqueName: \"kubernetes.io/projected/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-kube-api-access-9jf59\") pod \"maas-api-revoke-sub-b410d4a0-7tsxp\" (UID: \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:15.090505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:15.090308 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b410d4a0-7tsxp\" (UID: \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:15.091010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:15.090980 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b410d4a0-7tsxp\" (UID: \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:15.092838 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:15.092815 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b410d4a0-7tsxp\" (UID: \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:15.117973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:15.117925 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jf59\" (UniqueName: \"kubernetes.io/projected/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-kube-api-access-9jf59\") pod \"maas-api-revoke-sub-b410d4a0-7tsxp\" (UID: \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:15.240184 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:15.240100 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:15.486072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:15.486036 4883 generic.go:356] "Generic (PLEG): container finished" podID="20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0" containerID="950127894fe22f35fa3f6b81983def9c19b0ef6b3ae85354782ecd144b1946e3" exitCode=0 Oct 30 18:26:15.486479 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:15.486134 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" event={"ID":"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0","Type":"ContainerDied","Data":"950127894fe22f35fa3f6b81983def9c19b0ef6b3ae85354782ecd144b1946e3"} Oct 30 18:26:15.557889 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:15.557449 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp"] Oct 30 18:26:16.545447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:16.541358 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" event={"ID":"64d6d4e5-8996-4e7c-ad42-1eeff776df7f","Type":"ContainerStarted","Data":"37b6a2b72159da4d068431f4618e761d3babb33325bdc348496643ffdc685565"} Oct 30 18:26:16.545447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:16.541445 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" event={"ID":"64d6d4e5-8996-4e7c-ad42-1eeff776df7f","Type":"ContainerStarted","Data":"c8a448ad18102c4a2eccd10d5662c61f796dc9e6f52488d0958a9999c84f7ecd"} Oct 30 18:26:16.584165 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:16.580290 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" podStartSLOduration=2.580270087 podStartE2EDuration="2.580270087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:16.578776683 +0000 UTC m=+2035.535780520" watchObservedRunningTime="2026-10-30 18:26:16.580270087 +0000 UTC m=+2035.537273924" Oct 30 18:26:16.989870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:16.989843 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:17.030261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.030230 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-xks-ca\") pod \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\" (UID: \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\") " Oct 30 18:26:17.030465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.030289 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-openshift-ca\") pod \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\" (UID: \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\") " Oct 30 18:26:17.030465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.030374 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-kube-api-access-66wv9\" (UniqueName: \"kubernetes.io/projected/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-kube-api-access-66wv9\") pod \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\" (UID: \"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0\") " Oct 30 18:26:17.030777 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.030749 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-openshift-ca" pod "20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0" (UID: "20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:17.031066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.031001 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:17.040556 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.040519 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-xks-ca" pod "20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0" (UID: "20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:17.045585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.045271 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-kube-api-access-66wv9" pod "20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0" (UID: "20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0"). InnerVolumeSpecName "kube-api-access-66wv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:17.100927 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.100889 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w"] Oct 30 18:26:17.106661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.106361 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca5f33-dpv8w"] Oct 30 18:26:17.133017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.132899 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66wv9\" (UniqueName: \"kubernetes.io/projected/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-kube-api-access-66wv9\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:17.133017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.132929 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:17.182914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.182881 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db71944-6f08-407e-9ee8-bbb39cc0e190" path="/var/lib/kubelet/pods/9db71944-6f08-407e-9ee8-bbb39cc0e190/volumes" Oct 30 18:26:17.605360 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.605322 4883 generic.go:356] "Generic (PLEG): container finished" podID="64d6d4e5-8996-4e7c-ad42-1eeff776df7f" containerID="37b6a2b72159da4d068431f4618e761d3babb33325bdc348496643ffdc685565" exitCode=0 Oct 30 18:26:17.605883 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.605529 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" event={"ID":"64d6d4e5-8996-4e7c-ad42-1eeff776df7f","Type":"ContainerDied","Data":"37b6a2b72159da4d068431f4618e761d3babb33325bdc348496643ffdc685565"} Oct 30 18:26:17.611985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.607746 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" event={"ID":"20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0","Type":"ContainerDied","Data":"ebd16f8d4b8c98147806586951799e321c162cb06e7e8089ff6a8e5183951624"} Oct 30 18:26:17.611985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.607796 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz" Oct 30 18:26:17.611985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:17.607801 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd16f8d4b8c98147806586951799e321c162cb06e7e8089ff6a8e5183951624" Oct 30 18:26:18.787400 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:18.787365 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:18.862854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:18.862817 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-kube-api-access-9jf59\" (UniqueName: \"kubernetes.io/projected/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-kube-api-access-9jf59\") pod \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\" (UID: \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\") " Oct 30 18:26:18.863081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:18.863026 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-openshift-ca\") pod \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\" (UID: \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\") " Oct 30 18:26:18.863162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:18.863130 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-xks-ca\") pod \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\" (UID: \"64d6d4e5-8996-4e7c-ad42-1eeff776df7f\") " Oct 30 18:26:18.863522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:18.863453 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-openshift-ca" pod "64d6d4e5-8996-4e7c-ad42-1eeff776df7f" (UID: "64d6d4e5-8996-4e7c-ad42-1eeff776df7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:18.863778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:18.863755 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:18.866041 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:18.865984 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-kube-api-access-9jf59" pod "64d6d4e5-8996-4e7c-ad42-1eeff776df7f" (UID: "64d6d4e5-8996-4e7c-ad42-1eeff776df7f"). InnerVolumeSpecName "kube-api-access-9jf59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:18.866156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:18.866089 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-xks-ca" pod "64d6d4e5-8996-4e7c-ad42-1eeff776df7f" (UID: "64d6d4e5-8996-4e7c-ad42-1eeff776df7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:18.965394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:18.965354 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:18.965394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:18.965391 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jf59\" (UniqueName: \"kubernetes.io/projected/64d6d4e5-8996-4e7c-ad42-1eeff776df7f-kube-api-access-9jf59\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:19.635043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:19.635013 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" Oct 30 18:26:19.635197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:19.635040 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp" event={"ID":"64d6d4e5-8996-4e7c-ad42-1eeff776df7f","Type":"ContainerDied","Data":"c8a448ad18102c4a2eccd10d5662c61f796dc9e6f52488d0958a9999c84f7ecd"} Oct 30 18:26:19.635197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:19.635092 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8a448ad18102c4a2eccd10d5662c61f796dc9e6f52488d0958a9999c84f7ecd" Oct 30 18:26:25.814490 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:25.814449 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78"] Oct 30 18:26:25.815296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:25.815221 4883 cpu_manager.go:414] "containerMap: removing container" podUID="64d6d4e5-8996-4e7c-ad42-1eeff776df7f" containerName="revoke-keys" Oct 30 18:26:25.815296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:25.815241 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d6d4e5-8996-4e7c-ad42-1eeff776df7f" containerName="revoke-keys" Oct 30 18:26:25.815296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:25.815262 4883 cpu_manager.go:414] "containerMap: removing container" podUID="20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0" containerName="revoke-keys" Oct 30 18:26:25.815296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:25.815267 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0" containerName="revoke-keys" Oct 30 18:26:25.815533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:25.815411 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d6d4e5-8996-4e7c-ad42-1eeff776df7f" containerName="revoke-keys" Oct 30 18:26:25.815533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:25.815425 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0" containerName="revoke-keys" Oct 30 18:26:25.820694 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:25.820665 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:25.850423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:25.849946 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78"] Oct 30 18:26:26.008495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.008455 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.008669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.008558 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.008669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.008589 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48259124-69a9-4121-91cb-337d2724acb9-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.008754 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.008742 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.008800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.008763 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.008800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.008785 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.008969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.008951 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvz79\" (UniqueName: \"kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-kube-api-access-gvz79\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.009049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.009032 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48259124-69a9-4121-91cb-337d2724acb9-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.009120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.009104 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.110622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.110580 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.110622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.110624 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.110859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.110646 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.110859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.110695 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvz79\" (UniqueName: \"kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-kube-api-access-gvz79\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.110859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.110726 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48259124-69a9-4121-91cb-337d2724acb9-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.110859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.110765 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.110859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.110811 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.110859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.110856 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.111214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.110882 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48259124-69a9-4121-91cb-337d2724acb9-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.111214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.111130 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.111333 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.111254 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.111429 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.111403 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.111495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.111476 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.111532 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.111517 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48259124-69a9-4121-91cb-337d2724acb9-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.113044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.113018 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.113326 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.113307 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48259124-69a9-4121-91cb-337d2724acb9-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.135702 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.135675 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.135884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.135866 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvz79\" (UniqueName: \"kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-kube-api-access-gvz79\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.136659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.136639 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:26.463991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.463965 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78"] Oct 30 18:26:26.465678 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:26:26.465651 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48259124_69a9_4121_91cb_337d2724acb9.slice/crio-c17fd83a78cc7edcc738d40a42351b439c934211a3f1b422225ca456c938e81b WatchSource:0}: Error finding container c17fd83a78cc7edcc738d40a42351b439c934211a3f1b422225ca456c938e81b: Status 404 returned error can't find the container with id c17fd83a78cc7edcc738d40a42351b439c934211a3f1b422225ca456c938e81b Oct 30 18:26:26.468146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.468107 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:26:26.468248 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.468201 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:26:26.468248 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.468234 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:26:26.757037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.755064 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" event={"ID":"48259124-69a9-4121-91cb-337d2724acb9","Type":"ContainerStarted","Data":"24f4d64d7f2f2025df2aeaeb7e5a7f5bf5eb2d3824429a40e56b78f73e1e8a16"} Oct 30 18:26:26.757037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.755133 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" event={"ID":"48259124-69a9-4121-91cb-337d2724acb9","Type":"ContainerStarted","Data":"c17fd83a78cc7edcc738d40a42351b439c934211a3f1b422225ca456c938e81b"} Oct 30 18:26:26.849273 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:26.849198 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" podStartSLOduration=1.849176163 podStartE2EDuration="1.849176163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:26.830543948 +0000 UTC m=+2045.787547789" watchObservedRunningTime="2026-10-30 18:26:26.849176163 +0000 UTC m=+2045.806180000" Oct 30 18:26:27.137537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:27.137504 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:27.138949 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:27.138907 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:26:27.139054 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:27.138997 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" podUID="48259124-69a9-4121-91cb-337d2724acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 18:26:28.137862 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:28.137821 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:26:28.138318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:28.137879 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" podUID="48259124-69a9-4121-91cb-337d2724acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 18:26:29.137509 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.137473 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:26:29.137769 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.137531 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" podUID="48259124-69a9-4121-91cb-337d2724acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 18:26:29.743080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.743041 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z"] Oct 30 18:26:29.748592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.748567 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:29.781065 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.780784 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z"] Oct 30 18:26:29.781065 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.780916 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:26:29.781065 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.780990 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:26:29.795862 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.795697 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:26:29.795862 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.795827 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:26:29.810166 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.801927 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z\" (UID: \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:29.810166 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.802008 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv7b5\" (UniqueName: \"kubernetes.io/projected/043e3657-f6d6-45aa-9f5b-906c6705ce3a-kube-api-access-sv7b5\") pod \"maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z\" (UID: \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:29.810166 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.802179 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z\" (UID: \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:29.918956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.917217 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z\" (UID: \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:29.918956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.917292 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv7b5\" (UniqueName: \"kubernetes.io/projected/043e3657-f6d6-45aa-9f5b-906c6705ce3a-kube-api-access-sv7b5\") pod \"maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z\" (UID: \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:29.918956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:29.917509 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z\" (UID: \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:30.111500 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:30.028422 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z\" (UID: \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:30.111500 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:30.070541 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z\" (UID: \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:30.155553 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:30.155467 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:26:30.155753 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:30.155599 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" podUID="48259124-69a9-4121-91cb-337d2724acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 18:26:30.188911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:30.188672 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv7b5\" (UniqueName: \"kubernetes.io/projected/043e3657-f6d6-45aa-9f5b-906c6705ce3a-kube-api-access-sv7b5\") pod \"maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z\" (UID: \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:30.372035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:30.370196 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:30.827611 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:30.827496 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z"] Oct 30 18:26:30.827611 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:26:30.827548 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043e3657_f6d6_45aa_9f5b_906c6705ce3a.slice/crio-9f7740525178d2cadb8e71b91e68ebea2fd72e477ca6a2164f7d7882d8280e88 WatchSource:0}: Error finding container 9f7740525178d2cadb8e71b91e68ebea2fd72e477ca6a2164f7d7882d8280e88: Status 404 returned error can't find the container with id 9f7740525178d2cadb8e71b91e68ebea2fd72e477ca6a2164f7d7882d8280e88 Oct 30 18:26:30.899961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:30.899497 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" event={"ID":"043e3657-f6d6-45aa-9f5b-906c6705ce3a","Type":"ContainerStarted","Data":"9f7740525178d2cadb8e71b91e68ebea2fd72e477ca6a2164f7d7882d8280e88"} Oct 30 18:26:31.137773 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:31.137732 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:26:31.137973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:31.137801 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" podUID="48259124-69a9-4121-91cb-337d2724acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 18:26:31.908708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:31.908676 4883 generic.go:356] "Generic (PLEG): container finished" podID="043e3657-f6d6-45aa-9f5b-906c6705ce3a" containerID="fbc0d53ee178b43fa45be3d09a4cff7174613c5b811929526a266c0548048068" exitCode=0 Oct 30 18:26:31.909127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:31.908751 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" event={"ID":"043e3657-f6d6-45aa-9f5b-906c6705ce3a","Type":"ContainerDied","Data":"fbc0d53ee178b43fa45be3d09a4cff7174613c5b811929526a266c0548048068"} Oct 30 18:26:32.137774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:32.137688 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:26:32.137774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:32.137751 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" podUID="48259124-69a9-4121-91cb-337d2724acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 18:26:33.106620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.106594 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:33.141601 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.141563 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:33.141892 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.141868 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:33.143202 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.143182 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:26:33.238558 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.238473 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-xks-ca\") pod \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\" (UID: \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\") " Oct 30 18:26:33.238730 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.238633 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-openshift-ca\") pod \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\" (UID: \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\") " Oct 30 18:26:33.238794 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.238730 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043e3657-f6d6-45aa-9f5b-906c6705ce3a-kube-api-access-sv7b5\" (UniqueName: \"kubernetes.io/projected/043e3657-f6d6-45aa-9f5b-906c6705ce3a-kube-api-access-sv7b5\") pod \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\" (UID: \"043e3657-f6d6-45aa-9f5b-906c6705ce3a\") " Oct 30 18:26:33.239211 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.239179 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-openshift-ca" pod "043e3657-f6d6-45aa-9f5b-906c6705ce3a" (UID: "043e3657-f6d6-45aa-9f5b-906c6705ce3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:33.239993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.239967 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:33.243741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.242218 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-xks-ca" pod "043e3657-f6d6-45aa-9f5b-906c6705ce3a" (UID: "043e3657-f6d6-45aa-9f5b-906c6705ce3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:33.244116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.244058 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043e3657-f6d6-45aa-9f5b-906c6705ce3a-kube-api-access-sv7b5" pod "043e3657-f6d6-45aa-9f5b-906c6705ce3a" (UID: "043e3657-f6d6-45aa-9f5b-906c6705ce3a"). InnerVolumeSpecName "kube-api-access-sv7b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:33.341060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.341028 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043e3657-f6d6-45aa-9f5b-906c6705ce3a-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:33.341060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.341055 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv7b5\" (UniqueName: \"kubernetes.io/projected/043e3657-f6d6-45aa-9f5b-906c6705ce3a-kube-api-access-sv7b5\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:33.928078 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.928046 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" Oct 30 18:26:33.928264 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.928031 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z" event={"ID":"043e3657-f6d6-45aa-9f5b-906c6705ce3a","Type":"ContainerDied","Data":"9f7740525178d2cadb8e71b91e68ebea2fd72e477ca6a2164f7d7882d8280e88"} Oct 30 18:26:33.928264 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:33.928154 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f7740525178d2cadb8e71b91e68ebea2fd72e477ca6a2164f7d7882d8280e88" Oct 30 18:26:35.081588 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:26:35.081551 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:26:36.821650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.821611 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh"] Oct 30 18:26:36.822253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.822235 4883 cpu_manager.go:414] "containerMap: removing container" podUID="043e3657-f6d6-45aa-9f5b-906c6705ce3a" containerName="revoke-keys" Oct 30 18:26:36.822310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.822254 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043e3657-f6d6-45aa-9f5b-906c6705ce3a" containerName="revoke-keys" Oct 30 18:26:36.822385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.822375 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="043e3657-f6d6-45aa-9f5b-906c6705ce3a" containerName="revoke-keys" Oct 30 18:26:36.826181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.826159 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:36.845677 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.845648 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh"] Oct 30 18:26:36.855371 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.855339 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:26:36.855579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.855344 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:26:36.855579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.855347 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:26:36.855720 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.855426 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:26:36.907271 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.907238 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77f47639-xdmqh\" (UID: \"e3184250-2320-4ab5-a678-d96b88c9333d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:36.907271 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.907276 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjbhh\" (UniqueName: \"kubernetes.io/projected/e3184250-2320-4ab5-a678-d96b88c9333d-kube-api-access-kjbhh\") pod \"maas-api-revoke-sub-77f47639-xdmqh\" (UID: \"e3184250-2320-4ab5-a678-d96b88c9333d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:36.907542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:36.907499 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77f47639-xdmqh\" (UID: \"e3184250-2320-4ab5-a678-d96b88c9333d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:37.009588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.009550 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77f47639-xdmqh\" (UID: \"e3184250-2320-4ab5-a678-d96b88c9333d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:37.009789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.009729 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77f47639-xdmqh\" (UID: \"e3184250-2320-4ab5-a678-d96b88c9333d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:37.009789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.009756 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjbhh\" (UniqueName: \"kubernetes.io/projected/e3184250-2320-4ab5-a678-d96b88c9333d-kube-api-access-kjbhh\") pod \"maas-api-revoke-sub-77f47639-xdmqh\" (UID: \"e3184250-2320-4ab5-a678-d96b88c9333d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:37.010364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.010340 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77f47639-xdmqh\" (UID: \"e3184250-2320-4ab5-a678-d96b88c9333d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:37.012175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.012156 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77f47639-xdmqh\" (UID: \"e3184250-2320-4ab5-a678-d96b88c9333d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:37.033946 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.033903 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjbhh\" (UniqueName: \"kubernetes.io/projected/e3184250-2320-4ab5-a678-d96b88c9333d-kube-api-access-kjbhh\") pod \"maas-api-revoke-sub-77f47639-xdmqh\" (UID: \"e3184250-2320-4ab5-a678-d96b88c9333d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:37.139714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.139672 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:37.474501 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.474472 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh"] Oct 30 18:26:37.476477 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:26:37.476449 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3184250_2320_4ab5_a678_d96b88c9333d.slice/crio-41bff700cb2aa3ccbdd3ff1c5fd314914bc7a9b0ba701eca22daf51fe2a64334 WatchSource:0}: Error finding container 41bff700cb2aa3ccbdd3ff1c5fd314914bc7a9b0ba701eca22daf51fe2a64334: Status 404 returned error can't find the container with id 41bff700cb2aa3ccbdd3ff1c5fd314914bc7a9b0ba701eca22daf51fe2a64334 Oct 30 18:26:37.762192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.762106 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t"] Oct 30 18:26:37.767533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.767509 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:37.791358 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.791329 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t"] Oct 30 18:26:37.820652 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.820602 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6fk2\" (UniqueName: \"kubernetes.io/projected/cf54ed9d-3143-422f-82ac-349c2e32b3e1-kube-api-access-k6fk2\") pod \"maas-api-revoke-sub-332a9e40-9mx4t\" (UID: \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:37.820865 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.820670 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-332a9e40-9mx4t\" (UID: \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:37.820865 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.820723 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-332a9e40-9mx4t\" (UID: \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:37.932368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.922951 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-332a9e40-9mx4t\" (UID: \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:37.932368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.923108 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-332a9e40-9mx4t\" (UID: \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:37.947958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.937151 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6fk2\" (UniqueName: \"kubernetes.io/projected/cf54ed9d-3143-422f-82ac-349c2e32b3e1-kube-api-access-k6fk2\") pod \"maas-api-revoke-sub-332a9e40-9mx4t\" (UID: \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:37.947958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.938878 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-332a9e40-9mx4t\" (UID: \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:37.962972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:37.962478 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-332a9e40-9mx4t\" (UID: \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:38.066012 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:38.055220 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6fk2\" (UniqueName: \"kubernetes.io/projected/cf54ed9d-3143-422f-82ac-349c2e32b3e1-kube-api-access-k6fk2\") pod \"maas-api-revoke-sub-332a9e40-9mx4t\" (UID: \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:38.078778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:38.078731 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" event={"ID":"e3184250-2320-4ab5-a678-d96b88c9333d","Type":"ContainerStarted","Data":"a5879f9599d141a7855ce2a0b67e80979d10a860f4de4f68b9ab36c7964e4ba2"} Oct 30 18:26:38.079030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:38.078844 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" event={"ID":"e3184250-2320-4ab5-a678-d96b88c9333d","Type":"ContainerStarted","Data":"41bff700cb2aa3ccbdd3ff1c5fd314914bc7a9b0ba701eca22daf51fe2a64334"} Oct 30 18:26:38.089238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:38.089204 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:38.124081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:38.124022 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" podStartSLOduration=2.124006298 podStartE2EDuration="2.124006298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:38.123746052 +0000 UTC m=+2057.080749888" watchObservedRunningTime="2026-10-30 18:26:38.124006298 +0000 UTC m=+2057.081010127" Oct 30 18:26:38.414956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:38.414913 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t"] Oct 30 18:26:38.416976 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:26:38.416944 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf54ed9d_3143_422f_82ac_349c2e32b3e1.slice/crio-be3e4a143d85514bff177df378f382e2668bdc14944f910fb1fab6ec3b74f073 WatchSource:0}: Error finding container be3e4a143d85514bff177df378f382e2668bdc14944f910fb1fab6ec3b74f073: Status 404 returned error can't find the container with id be3e4a143d85514bff177df378f382e2668bdc14944f910fb1fab6ec3b74f073 Oct 30 18:26:39.087235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:39.087203 4883 generic.go:356] "Generic (PLEG): container finished" podID="e3184250-2320-4ab5-a678-d96b88c9333d" containerID="a5879f9599d141a7855ce2a0b67e80979d10a860f4de4f68b9ab36c7964e4ba2" exitCode=0 Oct 30 18:26:39.087615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:39.087288 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" event={"ID":"e3184250-2320-4ab5-a678-d96b88c9333d","Type":"ContainerDied","Data":"a5879f9599d141a7855ce2a0b67e80979d10a860f4de4f68b9ab36c7964e4ba2"} Oct 30 18:26:39.088897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:39.088865 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" event={"ID":"cf54ed9d-3143-422f-82ac-349c2e32b3e1","Type":"ContainerStarted","Data":"2fff9c25b4160383bd8543bbec346bb96b439b650a89bd46eb90e30ebeb78ff2"} Oct 30 18:26:39.089032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:39.088913 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" event={"ID":"cf54ed9d-3143-422f-82ac-349c2e32b3e1","Type":"ContainerStarted","Data":"be3e4a143d85514bff177df378f382e2668bdc14944f910fb1fab6ec3b74f073"} Oct 30 18:26:39.232974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:39.232832 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" podStartSLOduration=2.232812575 podStartE2EDuration="2.232812575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:39.184349542 +0000 UTC m=+2058.141353379" watchObservedRunningTime="2026-10-30 18:26:39.232812575 +0000 UTC m=+2058.189816412" Oct 30 18:26:40.098229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:40.098198 4883 generic.go:356] "Generic (PLEG): container finished" podID="cf54ed9d-3143-422f-82ac-349c2e32b3e1" containerID="2fff9c25b4160383bd8543bbec346bb96b439b650a89bd46eb90e30ebeb78ff2" exitCode=0 Oct 30 18:26:40.098781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:40.098276 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" event={"ID":"cf54ed9d-3143-422f-82ac-349c2e32b3e1","Type":"ContainerDied","Data":"2fff9c25b4160383bd8543bbec346bb96b439b650a89bd46eb90e30ebeb78ff2"} Oct 30 18:26:40.281390 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:40.281362 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:40.436405 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:40.436318 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-xks-ca\") pod \"e3184250-2320-4ab5-a678-d96b88c9333d\" (UID: \"e3184250-2320-4ab5-a678-d96b88c9333d\") " Oct 30 18:26:40.436581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:40.436432 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3184250-2320-4ab5-a678-d96b88c9333d-kube-api-access-kjbhh\" (UniqueName: \"kubernetes.io/projected/e3184250-2320-4ab5-a678-d96b88c9333d-kube-api-access-kjbhh\") pod \"e3184250-2320-4ab5-a678-d96b88c9333d\" (UID: \"e3184250-2320-4ab5-a678-d96b88c9333d\") " Oct 30 18:26:40.436581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:40.436539 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-openshift-ca\") pod \"e3184250-2320-4ab5-a678-d96b88c9333d\" (UID: \"e3184250-2320-4ab5-a678-d96b88c9333d\") " Oct 30 18:26:40.436905 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:40.436877 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-openshift-ca" pod "e3184250-2320-4ab5-a678-d96b88c9333d" (UID: "e3184250-2320-4ab5-a678-d96b88c9333d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:40.437117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:40.437100 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:40.438471 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:40.438448 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-xks-ca" pod "e3184250-2320-4ab5-a678-d96b88c9333d" (UID: "e3184250-2320-4ab5-a678-d96b88c9333d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:40.438561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:40.438534 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3184250-2320-4ab5-a678-d96b88c9333d-kube-api-access-kjbhh" pod "e3184250-2320-4ab5-a678-d96b88c9333d" (UID: "e3184250-2320-4ab5-a678-d96b88c9333d"). InnerVolumeSpecName "kube-api-access-kjbhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:40.539388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:40.539172 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3184250-2320-4ab5-a678-d96b88c9333d-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:40.539388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:40.539205 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjbhh\" (UniqueName: \"kubernetes.io/projected/e3184250-2320-4ab5-a678-d96b88c9333d-kube-api-access-kjbhh\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:41.115452 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.115416 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" Oct 30 18:26:41.115983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.115450 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh" event={"ID":"e3184250-2320-4ab5-a678-d96b88c9333d","Type":"ContainerDied","Data":"41bff700cb2aa3ccbdd3ff1c5fd314914bc7a9b0ba701eca22daf51fe2a64334"} Oct 30 18:26:41.115983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.115512 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41bff700cb2aa3ccbdd3ff1c5fd314914bc7a9b0ba701eca22daf51fe2a64334" Oct 30 18:26:41.306209 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.306186 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:41.366570 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.366488 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-openshift-ca\") pod \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\" (UID: \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\") " Oct 30 18:26:41.366725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.366571 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf54ed9d-3143-422f-82ac-349c2e32b3e1-kube-api-access-k6fk2\" (UniqueName: \"kubernetes.io/projected/cf54ed9d-3143-422f-82ac-349c2e32b3e1-kube-api-access-k6fk2\") pod \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\" (UID: \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\") " Oct 30 18:26:41.366725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.366645 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-xks-ca\") pod \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\" (UID: \"cf54ed9d-3143-422f-82ac-349c2e32b3e1\") " Oct 30 18:26:41.366910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.366864 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-openshift-ca" pod "cf54ed9d-3143-422f-82ac-349c2e32b3e1" (UID: "cf54ed9d-3143-422f-82ac-349c2e32b3e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:41.368079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.368053 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:41.370536 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.370505 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf54ed9d-3143-422f-82ac-349c2e32b3e1-kube-api-access-k6fk2" pod "cf54ed9d-3143-422f-82ac-349c2e32b3e1" (UID: "cf54ed9d-3143-422f-82ac-349c2e32b3e1"). InnerVolumeSpecName "kube-api-access-k6fk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:41.379474 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.379439 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-xks-ca" pod "cf54ed9d-3143-422f-82ac-349c2e32b3e1" (UID: "cf54ed9d-3143-422f-82ac-349c2e32b3e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:41.470099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.470061 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6fk2\" (UniqueName: \"kubernetes.io/projected/cf54ed9d-3143-422f-82ac-349c2e32b3e1-kube-api-access-k6fk2\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:41.470099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:41.470094 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf54ed9d-3143-422f-82ac-349c2e32b3e1-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:42.123608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:42.123565 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" event={"ID":"cf54ed9d-3143-422f-82ac-349c2e32b3e1","Type":"ContainerDied","Data":"be3e4a143d85514bff177df378f382e2668bdc14944f910fb1fab6ec3b74f073"} Oct 30 18:26:42.123608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:42.123592 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t" Oct 30 18:26:42.124047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:42.123650 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be3e4a143d85514bff177df378f382e2668bdc14944f910fb1fab6ec3b74f073" Oct 30 18:26:45.866243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.866208 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl"] Oct 30 18:26:45.866946 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.866919 4883 cpu_manager.go:414] "containerMap: removing container" podUID="e3184250-2320-4ab5-a678-d96b88c9333d" containerName="revoke-keys" Oct 30 18:26:45.867001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.866955 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3184250-2320-4ab5-a678-d96b88c9333d" containerName="revoke-keys" Oct 30 18:26:45.867001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.866990 4883 cpu_manager.go:414] "containerMap: removing container" podUID="cf54ed9d-3143-422f-82ac-349c2e32b3e1" containerName="revoke-keys" Oct 30 18:26:45.867001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.866995 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf54ed9d-3143-422f-82ac-349c2e32b3e1" containerName="revoke-keys" Oct 30 18:26:45.867117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.867106 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3184250-2320-4ab5-a678-d96b88c9333d" containerName="revoke-keys" Oct 30 18:26:45.867150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.867118 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf54ed9d-3143-422f-82ac-349c2e32b3e1" containerName="revoke-keys" Oct 30 18:26:45.870880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.870855 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:45.915909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.915878 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:26:45.921119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.921082 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:26:45.921265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.921139 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:26:45.921265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.921171 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:26:45.925324 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.925295 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl"] Oct 30 18:26:45.938693 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.938662 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl\" (UID: \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:45.938894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.938875 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbbpz\" (UniqueName: \"kubernetes.io/projected/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-kube-api-access-rbbpz\") pod \"maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl\" (UID: \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:45.939000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:45.938903 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl\" (UID: \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:46.039799 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:46.039767 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbbpz\" (UniqueName: \"kubernetes.io/projected/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-kube-api-access-rbbpz\") pod \"maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl\" (UID: \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:46.039799 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:46.039799 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl\" (UID: \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:46.040104 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:46.039894 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl\" (UID: \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:46.040668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:46.040646 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl\" (UID: \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:46.042247 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:46.042228 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl\" (UID: \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:46.064425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:46.064394 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbbpz\" (UniqueName: \"kubernetes.io/projected/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-kube-api-access-rbbpz\") pod \"maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl\" (UID: \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:46.196921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:46.196841 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:46.506092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:46.506066 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl"] Oct 30 18:26:46.508372 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:26:46.508340 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeceb0a6c_f56d_4c66_aa42_c9c3b17ff866.slice/crio-31a834df845b9af1527e7340aee418e92e7e84aca0905a184217476fc8d0daee WatchSource:0}: Error finding container 31a834df845b9af1527e7340aee418e92e7e84aca0905a184217476fc8d0daee: Status 404 returned error can't find the container with id 31a834df845b9af1527e7340aee418e92e7e84aca0905a184217476fc8d0daee Oct 30 18:26:47.183912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.183314 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" event={"ID":"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866","Type":"ContainerStarted","Data":"e551080a7705efa936a32a2061887d67e5ed33609eccaff99aa00acffd41556e"} Oct 30 18:26:47.183912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.183405 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" event={"ID":"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866","Type":"ContainerStarted","Data":"31a834df845b9af1527e7340aee418e92e7e84aca0905a184217476fc8d0daee"} Oct 30 18:26:47.225118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.225058 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" podStartSLOduration=2.225040728 podStartE2EDuration="2.225040728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:47.224481945 +0000 UTC m=+2066.181485783" watchObservedRunningTime="2026-10-30 18:26:47.225040728 +0000 UTC m=+2066.182044567" Oct 30 18:26:47.248791 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.248743 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx"] Oct 30 18:26:47.254266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.254219 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:47.280674 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.280639 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx"] Oct 30 18:26:47.361800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.361764 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce9a51f9-678wx\" (UID: \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:47.362133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.361820 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce9a51f9-678wx\" (UID: \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:47.362133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.362041 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kzt\" (UniqueName: \"kubernetes.io/projected/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-kube-api-access-d9kzt\") pod \"maas-api-revoke-sub-ce9a51f9-678wx\" (UID: \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:47.463385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.463300 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce9a51f9-678wx\" (UID: \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:47.463523 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.463385 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce9a51f9-678wx\" (UID: \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:47.463523 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.463512 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kzt\" (UniqueName: \"kubernetes.io/projected/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-kube-api-access-d9kzt\") pod \"maas-api-revoke-sub-ce9a51f9-678wx\" (UID: \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:47.464055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.464028 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce9a51f9-678wx\" (UID: \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:47.465643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.465625 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce9a51f9-678wx\" (UID: \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:47.488317 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.488289 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kzt\" (UniqueName: \"kubernetes.io/projected/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-kube-api-access-d9kzt\") pod \"maas-api-revoke-sub-ce9a51f9-678wx\" (UID: \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:47.571721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.571686 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:47.882488 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:47.882466 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx"] Oct 30 18:26:47.884543 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:26:47.884514 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02aa44b5_7d4b_46c3_811a_5cca736fe4cb.slice/crio-61645a59d19d2dad4add458a54e069994821424e51cebd472d9d3e848561fad0 WatchSource:0}: Error finding container 61645a59d19d2dad4add458a54e069994821424e51cebd472d9d3e848561fad0: Status 404 returned error can't find the container with id 61645a59d19d2dad4add458a54e069994821424e51cebd472d9d3e848561fad0 Oct 30 18:26:48.189997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.189966 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" event={"ID":"02aa44b5-7d4b-46c3-811a-5cca736fe4cb","Type":"ContainerStarted","Data":"61645a59d19d2dad4add458a54e069994821424e51cebd472d9d3e848561fad0"} Oct 30 18:26:48.192022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.191995 4883 generic.go:356] "Generic (PLEG): container finished" podID="eceb0a6c-f56d-4c66-aa42-c9c3b17ff866" containerID="e551080a7705efa936a32a2061887d67e5ed33609eccaff99aa00acffd41556e" exitCode=0 Oct 30 18:26:48.192160 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.192089 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" event={"ID":"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866","Type":"ContainerDied","Data":"e551080a7705efa936a32a2061887d67e5ed33609eccaff99aa00acffd41556e"} Oct 30 18:26:48.244479 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.244435 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" podStartSLOduration=1.244422616 podStartE2EDuration="1.244422616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:48.241030433 +0000 UTC m=+2067.198034269" watchObservedRunningTime="2026-10-30 18:26:48.244422616 +0000 UTC m=+2067.201426453" Oct 30 18:26:48.404963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.403993 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6"] Oct 30 18:26:48.420007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.419173 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6"] Oct 30 18:26:48.420007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.419299 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:48.485410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.485369 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e35b3c8-gvwq6\" (UID: \"db09c429-cef5-4171-ad7e-8499fdf09288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:48.485726 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.485692 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d25g8\" (UniqueName: \"kubernetes.io/projected/db09c429-cef5-4171-ad7e-8499fdf09288-kube-api-access-d25g8\") pod \"maas-api-revoke-sub-4e35b3c8-gvwq6\" (UID: \"db09c429-cef5-4171-ad7e-8499fdf09288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:48.486031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.486008 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e35b3c8-gvwq6\" (UID: \"db09c429-cef5-4171-ad7e-8499fdf09288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:48.587712 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.587671 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d25g8\" (UniqueName: \"kubernetes.io/projected/db09c429-cef5-4171-ad7e-8499fdf09288-kube-api-access-d25g8\") pod \"maas-api-revoke-sub-4e35b3c8-gvwq6\" (UID: \"db09c429-cef5-4171-ad7e-8499fdf09288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:48.587910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.587767 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e35b3c8-gvwq6\" (UID: \"db09c429-cef5-4171-ad7e-8499fdf09288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:48.587910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.587863 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e35b3c8-gvwq6\" (UID: \"db09c429-cef5-4171-ad7e-8499fdf09288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:48.588530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.588506 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e35b3c8-gvwq6\" (UID: \"db09c429-cef5-4171-ad7e-8499fdf09288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:48.590284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.590261 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e35b3c8-gvwq6\" (UID: \"db09c429-cef5-4171-ad7e-8499fdf09288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:48.614064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.614038 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d25g8\" (UniqueName: \"kubernetes.io/projected/db09c429-cef5-4171-ad7e-8499fdf09288-kube-api-access-d25g8\") pod \"maas-api-revoke-sub-4e35b3c8-gvwq6\" (UID: \"db09c429-cef5-4171-ad7e-8499fdf09288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:48.761016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:48.760796 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:49.070668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.070631 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6"] Oct 30 18:26:49.071538 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:26:49.071508 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09c429_cef5_4171_ad7e_8499fdf09288.slice/crio-4b0f2d9a28543459211839f53ff05fb373e6804fc8ab4628331077deb691f58a WatchSource:0}: Error finding container 4b0f2d9a28543459211839f53ff05fb373e6804fc8ab4628331077deb691f58a: Status 404 returned error can't find the container with id 4b0f2d9a28543459211839f53ff05fb373e6804fc8ab4628331077deb691f58a Oct 30 18:26:49.202091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.202053 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" event={"ID":"db09c429-cef5-4171-ad7e-8499fdf09288","Type":"ContainerStarted","Data":"4b0f2d9a28543459211839f53ff05fb373e6804fc8ab4628331077deb691f58a"} Oct 30 18:26:49.203697 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.203669 4883 generic.go:356] "Generic (PLEG): container finished" podID="02aa44b5-7d4b-46c3-811a-5cca736fe4cb" containerID="7a49af92cd71a79abcfb320b6fb50a0f7f627e7542aa760a7b68c8a89905fc33" exitCode=0 Oct 30 18:26:49.203828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.203744 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" event={"ID":"02aa44b5-7d4b-46c3-811a-5cca736fe4cb","Type":"ContainerDied","Data":"7a49af92cd71a79abcfb320b6fb50a0f7f627e7542aa760a7b68c8a89905fc33"} Oct 30 18:26:49.454419 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.454372 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:49.501500 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.501463 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-openshift-ca\") pod \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\" (UID: \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\") " Oct 30 18:26:49.501669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.501644 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-xks-ca\") pod \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\" (UID: \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\") " Oct 30 18:26:49.501762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.501715 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-kube-api-access-rbbpz\" (UniqueName: \"kubernetes.io/projected/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-kube-api-access-rbbpz\") pod \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\" (UID: \"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866\") " Oct 30 18:26:49.501986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.501949 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-openshift-ca" pod "eceb0a6c-f56d-4c66-aa42-c9c3b17ff866" (UID: "eceb0a6c-f56d-4c66-aa42-c9c3b17ff866"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:49.502599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.502571 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:49.503912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.503878 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-xks-ca" pod "eceb0a6c-f56d-4c66-aa42-c9c3b17ff866" (UID: "eceb0a6c-f56d-4c66-aa42-c9c3b17ff866"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:49.504612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.504589 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-kube-api-access-rbbpz" pod "eceb0a6c-f56d-4c66-aa42-c9c3b17ff866" (UID: "eceb0a6c-f56d-4c66-aa42-c9c3b17ff866"). InnerVolumeSpecName "kube-api-access-rbbpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:49.604365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.604331 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:49.604365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:49.604360 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbbpz\" (UniqueName: \"kubernetes.io/projected/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866-kube-api-access-rbbpz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:50.212423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.212395 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" Oct 30 18:26:50.212858 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.212445 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl" event={"ID":"eceb0a6c-f56d-4c66-aa42-c9c3b17ff866","Type":"ContainerDied","Data":"31a834df845b9af1527e7340aee418e92e7e84aca0905a184217476fc8d0daee"} Oct 30 18:26:50.212858 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.212503 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31a834df845b9af1527e7340aee418e92e7e84aca0905a184217476fc8d0daee" Oct 30 18:26:50.214200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.214175 4883 generic.go:356] "Generic (PLEG): container finished" podID="db09c429-cef5-4171-ad7e-8499fdf09288" containerID="f10e81b745834f51445d17d00d5ae5e520045d9b0314608f50664684a2a0ae0e" exitCode=0 Oct 30 18:26:50.214334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.214310 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" event={"ID":"db09c429-cef5-4171-ad7e-8499fdf09288","Type":"ContainerDied","Data":"f10e81b745834f51445d17d00d5ae5e520045d9b0314608f50664684a2a0ae0e"} Oct 30 18:26:50.429630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.427717 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:50.522819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.520293 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-kube-api-access-d9kzt\" (UniqueName: \"kubernetes.io/projected/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-kube-api-access-d9kzt\") pod \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\" (UID: \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\") " Oct 30 18:26:50.522819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.520612 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-xks-ca\") pod \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\" (UID: \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\") " Oct 30 18:26:50.522819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.520671 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-openshift-ca\") pod \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\" (UID: \"02aa44b5-7d4b-46c3-811a-5cca736fe4cb\") " Oct 30 18:26:50.522819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.521790 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-openshift-ca" pod "02aa44b5-7d4b-46c3-811a-5cca736fe4cb" (UID: "02aa44b5-7d4b-46c3-811a-5cca736fe4cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:50.530036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.525643 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-xks-ca" pod "02aa44b5-7d4b-46c3-811a-5cca736fe4cb" (UID: "02aa44b5-7d4b-46c3-811a-5cca736fe4cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:50.530036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.526170 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-kube-api-access-d9kzt" pod "02aa44b5-7d4b-46c3-811a-5cca736fe4cb" (UID: "02aa44b5-7d4b-46c3-811a-5cca736fe4cb"). InnerVolumeSpecName "kube-api-access-d9kzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:50.623083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.623045 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:50.623083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.623085 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:50.623287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:50.623105 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9kzt\" (UniqueName: \"kubernetes.io/projected/02aa44b5-7d4b-46c3-811a-5cca736fe4cb-kube-api-access-d9kzt\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:51.224991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.224960 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" Oct 30 18:26:51.225486 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.224982 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx" event={"ID":"02aa44b5-7d4b-46c3-811a-5cca736fe4cb","Type":"ContainerDied","Data":"61645a59d19d2dad4add458a54e069994821424e51cebd472d9d3e848561fad0"} Oct 30 18:26:51.225642 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.225534 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61645a59d19d2dad4add458a54e069994821424e51cebd472d9d3e848561fad0" Oct 30 18:26:51.402972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.402947 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:51.459106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.453705 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-xks-ca\") pod \"db09c429-cef5-4171-ad7e-8499fdf09288\" (UID: \"db09c429-cef5-4171-ad7e-8499fdf09288\") " Oct 30 18:26:51.459106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.453757 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-openshift-ca\") pod \"db09c429-cef5-4171-ad7e-8499fdf09288\" (UID: \"db09c429-cef5-4171-ad7e-8499fdf09288\") " Oct 30 18:26:51.459106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.453860 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db09c429-cef5-4171-ad7e-8499fdf09288-kube-api-access-d25g8\" (UniqueName: \"kubernetes.io/projected/db09c429-cef5-4171-ad7e-8499fdf09288-kube-api-access-d25g8\") pod \"db09c429-cef5-4171-ad7e-8499fdf09288\" (UID: \"db09c429-cef5-4171-ad7e-8499fdf09288\") " Oct 30 18:26:51.459106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.455437 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-openshift-ca" pod "db09c429-cef5-4171-ad7e-8499fdf09288" (UID: "db09c429-cef5-4171-ad7e-8499fdf09288"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:51.459106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.456456 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-xks-ca" pod "db09c429-cef5-4171-ad7e-8499fdf09288" (UID: "db09c429-cef5-4171-ad7e-8499fdf09288"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:51.463032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.463000 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db09c429-cef5-4171-ad7e-8499fdf09288-kube-api-access-d25g8" pod "db09c429-cef5-4171-ad7e-8499fdf09288" (UID: "db09c429-cef5-4171-ad7e-8499fdf09288"). InnerVolumeSpecName "kube-api-access-d25g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:51.557084 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.557047 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d25g8\" (UniqueName: \"kubernetes.io/projected/db09c429-cef5-4171-ad7e-8499fdf09288-kube-api-access-d25g8\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:51.557084 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.557087 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:51.557084 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:51.557103 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db09c429-cef5-4171-ad7e-8499fdf09288-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:26:52.242314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:52.242276 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" Oct 30 18:26:52.242314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:52.242304 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6" event={"ID":"db09c429-cef5-4171-ad7e-8499fdf09288","Type":"ContainerDied","Data":"4b0f2d9a28543459211839f53ff05fb373e6804fc8ab4628331077deb691f58a"} Oct 30 18:26:52.242738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:52.242375 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b0f2d9a28543459211839f53ff05fb373e6804fc8ab4628331077deb691f58a" Oct 30 18:26:56.345053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:56.345011 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z"] Oct 30 18:26:56.354735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:56.354702 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c2b103-57f9z"] Oct 30 18:26:57.178522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:57.178489 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="043e3657-f6d6-45aa-9f5b-906c6705ce3a" path="/var/lib/kubelet/pods/043e3657-f6d6-45aa-9f5b-906c6705ce3a/volumes" Oct 30 18:26:59.214189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:59.214153 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl"] Oct 30 18:26:59.226617 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:26:59.226584 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-42fad7-pknnl"] Oct 30 18:27:00.129098 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.129064 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9"] Oct 30 18:27:00.129669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.129654 4883 cpu_manager.go:414] "containerMap: removing container" podUID="eceb0a6c-f56d-4c66-aa42-c9c3b17ff866" containerName="revoke-keys" Oct 30 18:27:00.129717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.129672 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eceb0a6c-f56d-4c66-aa42-c9c3b17ff866" containerName="revoke-keys" Oct 30 18:27:00.129717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.129681 4883 cpu_manager.go:414] "containerMap: removing container" podUID="02aa44b5-7d4b-46c3-811a-5cca736fe4cb" containerName="revoke-keys" Oct 30 18:27:00.129717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.129686 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02aa44b5-7d4b-46c3-811a-5cca736fe4cb" containerName="revoke-keys" Oct 30 18:27:00.129717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.129704 4883 cpu_manager.go:414] "containerMap: removing container" podUID="db09c429-cef5-4171-ad7e-8499fdf09288" containerName="revoke-keys" Oct 30 18:27:00.129717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.129710 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db09c429-cef5-4171-ad7e-8499fdf09288" containerName="revoke-keys" Oct 30 18:27:00.129893 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.129804 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="eceb0a6c-f56d-4c66-aa42-c9c3b17ff866" containerName="revoke-keys" Oct 30 18:27:00.129893 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.129813 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="02aa44b5-7d4b-46c3-811a-5cca736fe4cb" containerName="revoke-keys" Oct 30 18:27:00.129893 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.129823 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="db09c429-cef5-4171-ad7e-8499fdf09288" containerName="revoke-keys" Oct 30 18:27:00.134802 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.134783 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:00.153112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.153092 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9"] Oct 30 18:27:00.162242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.162211 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:27:00.162242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.162225 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:27:00.162422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.162279 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:00.164198 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.164174 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:27:00.215709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.215661 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e77bc3a-gbwr9\" (UID: \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:00.216362 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.215825 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e77bc3a-gbwr9\" (UID: \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:00.216362 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.215965 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9wtz\" (UniqueName: \"kubernetes.io/projected/3a40295a-67e7-4173-aca6-6354a8dd2fc6-kube-api-access-m9wtz\") pod \"maas-api-revoke-sub-4e77bc3a-gbwr9\" (UID: \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:00.317914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.317868 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e77bc3a-gbwr9\" (UID: \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:00.318136 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.317985 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e77bc3a-gbwr9\" (UID: \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:00.318136 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.318067 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wtz\" (UniqueName: \"kubernetes.io/projected/3a40295a-67e7-4173-aca6-6354a8dd2fc6-kube-api-access-m9wtz\") pod \"maas-api-revoke-sub-4e77bc3a-gbwr9\" (UID: \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:00.318621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.318597 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e77bc3a-gbwr9\" (UID: \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:00.320479 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.320458 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e77bc3a-gbwr9\" (UID: \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:00.343168 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.343141 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9wtz\" (UniqueName: \"kubernetes.io/projected/3a40295a-67e7-4173-aca6-6354a8dd2fc6-kube-api-access-m9wtz\") pod \"maas-api-revoke-sub-4e77bc3a-gbwr9\" (UID: \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:00.449242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.449164 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:00.777623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:00.772134 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9"] Oct 30 18:27:00.779490 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:27:00.779451 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a40295a_67e7_4173_aca6_6354a8dd2fc6.slice/crio-4e310c5d78b7aa98ad393c587006dc5fb688fd9dac178aeefa8b83909b2fe797 WatchSource:0}: Error finding container 4e310c5d78b7aa98ad393c587006dc5fb688fd9dac178aeefa8b83909b2fe797: Status 404 returned error can't find the container with id 4e310c5d78b7aa98ad393c587006dc5fb688fd9dac178aeefa8b83909b2fe797 Oct 30 18:27:01.184715 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:01.184681 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eceb0a6c-f56d-4c66-aa42-c9c3b17ff866" path="/var/lib/kubelet/pods/eceb0a6c-f56d-4c66-aa42-c9c3b17ff866/volumes" Oct 30 18:27:01.332773 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:01.332741 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" event={"ID":"3a40295a-67e7-4173-aca6-6354a8dd2fc6","Type":"ContainerStarted","Data":"7f31a1f5c3b74225444ab8ea32a3bf3a450dfae05a3ba51d3a3fa839bb2d5964"} Oct 30 18:27:01.333179 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:01.332793 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" event={"ID":"3a40295a-67e7-4173-aca6-6354a8dd2fc6","Type":"ContainerStarted","Data":"4e310c5d78b7aa98ad393c587006dc5fb688fd9dac178aeefa8b83909b2fe797"} Oct 30 18:27:01.376815 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:01.376767 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" podStartSLOduration=1.376754283 podStartE2EDuration="1.376754283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:01.375241652 +0000 UTC m=+2080.332245490" watchObservedRunningTime="2026-10-30 18:27:01.376754283 +0000 UTC m=+2080.333758134" Oct 30 18:27:02.341741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:02.341708 4883 generic.go:356] "Generic (PLEG): container finished" podID="3a40295a-67e7-4173-aca6-6354a8dd2fc6" containerID="7f31a1f5c3b74225444ab8ea32a3bf3a450dfae05a3ba51d3a3fa839bb2d5964" exitCode=0 Oct 30 18:27:02.342143 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:02.341796 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" event={"ID":"3a40295a-67e7-4173-aca6-6354a8dd2fc6","Type":"ContainerDied","Data":"7f31a1f5c3b74225444ab8ea32a3bf3a450dfae05a3ba51d3a3fa839bb2d5964"} Oct 30 18:27:03.742623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:03.742553 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:03.810956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:03.805825 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a40295a-67e7-4173-aca6-6354a8dd2fc6-kube-api-access-m9wtz\" (UniqueName: \"kubernetes.io/projected/3a40295a-67e7-4173-aca6-6354a8dd2fc6-kube-api-access-m9wtz\") pod \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\" (UID: \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\") " Oct 30 18:27:03.810956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:03.805890 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-xks-ca\") pod \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\" (UID: \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\") " Oct 30 18:27:03.810956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:03.805948 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-openshift-ca\") pod \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\" (UID: \"3a40295a-67e7-4173-aca6-6354a8dd2fc6\") " Oct 30 18:27:03.814521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:03.814479 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-openshift-ca" pod "3a40295a-67e7-4173-aca6-6354a8dd2fc6" (UID: "3a40295a-67e7-4173-aca6-6354a8dd2fc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:03.817434 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:03.817409 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:03.821644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:03.821598 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a40295a-67e7-4173-aca6-6354a8dd2fc6-kube-api-access-m9wtz" pod "3a40295a-67e7-4173-aca6-6354a8dd2fc6" (UID: "3a40295a-67e7-4173-aca6-6354a8dd2fc6"). InnerVolumeSpecName "kube-api-access-m9wtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:03.823571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:03.823545 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-xks-ca" pod "3a40295a-67e7-4173-aca6-6354a8dd2fc6" (UID: "3a40295a-67e7-4173-aca6-6354a8dd2fc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:03.920017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:03.919901 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9wtz\" (UniqueName: \"kubernetes.io/projected/3a40295a-67e7-4173-aca6-6354a8dd2fc6-kube-api-access-m9wtz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:03.920289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:03.920244 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a40295a-67e7-4173-aca6-6354a8dd2fc6-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:04.243117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:04.241643 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7"] Oct 30 18:27:04.243117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:04.241911 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" podUID="36fb3684-6cf3-4d2d-b930-ad0388420357" containerName="istio-proxy" containerID="cri-o://5dff6df0f2c7eac44e32d21ea545d3ea352421c1a597907131a95cae25324f5d" gracePeriod=30 Oct 30 18:27:04.371960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:04.352217 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c"] Oct 30 18:27:04.371960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:04.352496 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" podUID="7ca77461-d958-4246-8ca8-0c45925a9e6c" containerName="istio-proxy" containerID="cri-o://b371823883b2993929be1ebe01afe19177e8ed9ca53ce7bbcbec75745036038c" gracePeriod=30 Oct 30 18:27:04.678608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:04.678573 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" event={"ID":"3a40295a-67e7-4173-aca6-6354a8dd2fc6","Type":"ContainerDied","Data":"4e310c5d78b7aa98ad393c587006dc5fb688fd9dac178aeefa8b83909b2fe797"} Oct 30 18:27:04.678897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:04.678881 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e310c5d78b7aa98ad393c587006dc5fb688fd9dac178aeefa8b83909b2fe797" Oct 30 18:27:04.679201 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:04.679167 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9" Oct 30 18:27:05.318529 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:27:05.318485 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:27:06.218800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:06.214746 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks"] Oct 30 18:27:06.218800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:06.215032 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" podUID="549a0999-9c9c-414a-ab9b-8574a8660dbf" containerName="istio-proxy" containerID="cri-o://5785b57fb390aec729843b19a0fb7ef4211a37f267eea03f94a143b5d0a723b5" gracePeriod=30 Oct 30 18:27:06.552315 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:06.552210 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:27:06.582246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:06.582200 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:06.656792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:06.656748 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 30 18:27:06.744099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:06.744048 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:06.881056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:06.881008 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded" Oct 30 18:27:06.909959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:06.906049 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:27:06.944955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:06.944599 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.003235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.003187 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.038069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.038020 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.038646 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.038613 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:27:07.039090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.039066 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.053048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.053006 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.059032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.059002 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.416960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.398491 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.555162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.555099 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.556353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.556320 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:27:07.621114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.621067 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.621924 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.621903 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:27:07.749155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.747057 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.749155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.747681 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:27:07.926460 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.909046 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.926460 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.909607 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:27:07.960105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.952972 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:07.960105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:07.953546 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:27:08.011419 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.011326 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:08.120958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.012165 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:27:08.373028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.356059 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:27:08.373028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.044197 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:27:08.373028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.044232 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:27:08.373028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.044256 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:27:08.373028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.358670 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:27:08.373028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.358723 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:27:08.373028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.358761 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:27:08.373028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.118925 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:27:08.373028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.118988 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:27:08.373028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.198804 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Oct 30 18:27:08.376369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.376343 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:27:08.376563 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.376551 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:27:08.439376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.439331 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:08.502956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.490752 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:08.502956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.491713 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:27:08.733012 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.636675 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:27:08.782957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.765439 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:08.916972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.912044 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:08.960999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:08.957385 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:09.019720 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.019597 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:27:09.062400 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.047295 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:09.256067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.245249 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:09.333916 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.333814 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:27:09.363356 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.363304 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:27:09.365495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.365444 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:27:09.366040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.366006 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:27:09.367287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.367254 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:27:09.385328 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.385274 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:27:09.389586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.389378 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:09.495228 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.491240 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:09.768406 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.768353 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:09.769324 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.769292 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:27:09.769768 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.769741 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:09.831280 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:27:09.831238 4883 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca77461_d958_4246_8ca8_0c45925a9e6c.slice/crio-conmon-b371823883b2993929be1ebe01afe19177e8ed9ca53ce7bbcbec75745036038c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:27:09.889562 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:27:09.889523 4883 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca77461_d958_4246_8ca8_0c45925a9e6c.slice/crio-b371823883b2993929be1ebe01afe19177e8ed9ca53ce7bbcbec75745036038c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:27:09.944082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.944038 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:09.992259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:09.977913 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:27:10.064959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.030842 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Oct 30 18:27:10.064959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.061493 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:27:10.222634 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.222503 4883 generic.go:356] "Generic (PLEG): container finished" podID="36fb3684-6cf3-4d2d-b930-ad0388420357" containerID="5dff6df0f2c7eac44e32d21ea545d3ea352421c1a597907131a95cae25324f5d" exitCode=0 Oct 30 18:27:10.222962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.222915 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" event={"ID":"36fb3684-6cf3-4d2d-b930-ad0388420357","Type":"ContainerDied","Data":"5dff6df0f2c7eac44e32d21ea545d3ea352421c1a597907131a95cae25324f5d"} Oct 30 18:27:10.404967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.340988 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:27:10.404967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.349838 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:27:10.404967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.395852 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:10.404967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.396259 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:27:10.404967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.396548 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:27:10.404967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.396812 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:27:10.404967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.397114 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:27:10.517017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.508128 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:10.645091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.645033 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:10.718551 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.708612 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 18:27:10.831023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.829113 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:10.831023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.829614 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:27:10.945553 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.945033 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:27:10.988977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:10.988144 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.084021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.066101 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:27:11.385223 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.367631 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.429972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.426888 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.450962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.440318 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.450962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.441881 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:27:11.450962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.442133 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:27:11.450962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.442322 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.521973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.515110 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.686401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.634182 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh"] Oct 30 18:27:11.686401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.635142 4883 cpu_manager.go:414] "containerMap: removing container" podUID="3a40295a-67e7-4173-aca6-6354a8dd2fc6" containerName="revoke-keys" Oct 30 18:27:11.686401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.635162 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a40295a-67e7-4173-aca6-6354a8dd2fc6" containerName="revoke-keys" Oct 30 18:27:11.686401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.635348 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a40295a-67e7-4173-aca6-6354a8dd2fc6" containerName="revoke-keys" Oct 30 18:27:11.697037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.696966 4883 generic.go:356] "Generic (PLEG): container finished" podID="7ca77461-d958-4246-8ca8-0c45925a9e6c" containerID="b371823883b2993929be1ebe01afe19177e8ed9ca53ce7bbcbec75745036038c" exitCode=0 Oct 30 18:27:11.716220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.716185 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh"] Oct 30 18:27:11.726095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.726054 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" event={"ID":"7ca77461-d958-4246-8ca8-0c45925a9e6c","Type":"ContainerDied","Data":"b371823883b2993929be1ebe01afe19177e8ed9ca53ce7bbcbec75745036038c"} Oct 30 18:27:11.726341 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.717094 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:11.775334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.747226 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:11.775334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.749168 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:27:11.775334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.749390 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:27:11.775334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.749436 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:27:11.877688 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.832068 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.877688 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.832424 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.877688 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.834634 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n8pz\" (UniqueName: \"kubernetes.io/projected/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-kube-api-access-9n8pz\") pod \"maas-api-revoke-sub-0c6a443c-hvzkh\" (UID: \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:11.877688 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.834786 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6a443c-hvzkh\" (UID: \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:11.877688 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.834859 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6a443c-hvzkh\" (UID: \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:11.940021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.939912 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6a443c-hvzkh\" (UID: \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:11.940982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.940950 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8pz\" (UniqueName: \"kubernetes.io/projected/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-kube-api-access-9n8pz\") pod \"maas-api-revoke-sub-0c6a443c-hvzkh\" (UID: \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:11.941284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.941264 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6a443c-hvzkh\" (UID: \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:11.942244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.942220 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6a443c-hvzkh\" (UID: \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:11.946104 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.946071 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:11.955391 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:11.955367 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6a443c-hvzkh\" (UID: \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:12.018318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.001219 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:12.018318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.013849 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n8pz\" (UniqueName: \"kubernetes.io/projected/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-kube-api-access-9n8pz\") pod \"maas-api-revoke-sub-0c6a443c-hvzkh\" (UID: \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:12.069130 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.069081 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:12.100959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.099222 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:12.415189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.402313 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:27:12.442513 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.442464 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:27:12.444960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.443223 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:27:12.456549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.446080 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:27:12.457261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.446109 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:27:12.467957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.446150 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:27:12.526956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.516380 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:12.841263 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.835215 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:12.841263 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.839028 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:27:12.955281 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:12.955226 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:13.052412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.013274 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:27:13.074179 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.071982 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.107990 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.119017 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-credential-socket\") pod \"36fb3684-6cf3-4d2d-b930-ad0388420357\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.119080 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-data\") pod \"36fb3684-6cf3-4d2d-b930-ad0388420357\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.119184 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-kube-api-access-d6nq5\" (UniqueName: \"kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-kube-api-access-d6nq5\") pod \"36fb3684-6cf3-4d2d-b930-ad0388420357\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.119218 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-envoy\") pod \"36fb3684-6cf3-4d2d-b930-ad0388420357\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.119251 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-token\" (UniqueName: \"kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-token\") pod \"36fb3684-6cf3-4d2d-b930-ad0388420357\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.119314 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-certs\") pod \"36fb3684-6cf3-4d2d-b930-ad0388420357\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.119372 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36fb3684-6cf3-4d2d-b930-ad0388420357-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36fb3684-6cf3-4d2d-b930-ad0388420357-istiod-ca-cert\") pod \"36fb3684-6cf3-4d2d-b930-ad0388420357\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.119422 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-socket\") pod \"36fb3684-6cf3-4d2d-b930-ad0388420357\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.119517 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-podinfo\") pod \"36fb3684-6cf3-4d2d-b930-ad0388420357\" (UID: \"36fb3684-6cf3-4d2d-b930-ad0388420357\") " Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.121226 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-certs" pod "36fb3684-6cf3-4d2d-b930-ad0388420357" (UID: "36fb3684-6cf3-4d2d-b930-ad0388420357"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.122008 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-data" pod "36fb3684-6cf3-4d2d-b930-ad0388420357" (UID: "36fb3684-6cf3-4d2d-b930-ad0388420357"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.126013 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-credential-socket" pod "36fb3684-6cf3-4d2d-b930-ad0388420357" (UID: "36fb3684-6cf3-4d2d-b930-ad0388420357"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.142263 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-socket" pod "36fb3684-6cf3-4d2d-b930-ad0388420357" (UID: "36fb3684-6cf3-4d2d-b930-ad0388420357"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:13.156582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.142521 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36fb3684-6cf3-4d2d-b930-ad0388420357-istiod-ca-cert" pod "36fb3684-6cf3-4d2d-b930-ad0388420357" (UID: "36fb3684-6cf3-4d2d-b930-ad0388420357"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:13.336251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.335204 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.336251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.335245 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.336251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.335263 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.336251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.335276 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36fb3684-6cf3-4d2d-b930-ad0388420357-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.336251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.335290 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.405387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.405325 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:13.512066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.510525 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:13.512066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.510837 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-podinfo" pod "36fb3684-6cf3-4d2d-b930-ad0388420357" (UID: "36fb3684-6cf3-4d2d-b930-ad0388420357"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:13.512307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.512204 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.512307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.512254 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:27:13.512774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.512468 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:27:13.512774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.512647 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:27:13.519508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.514647 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-token" pod "36fb3684-6cf3-4d2d-b930-ad0388420357" (UID: "36fb3684-6cf3-4d2d-b930-ad0388420357"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:13.526829 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.525432 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:13.526829 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.525575 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:27:13.560975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.553590 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-kube-api-access-d6nq5" pod "36fb3684-6cf3-4d2d-b930-ad0388420357" (UID: "36fb3684-6cf3-4d2d-b930-ad0388420357"). InnerVolumeSpecName "kube-api-access-d6nq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:13.581592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.581180 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:13.581592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.581233 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:13.595992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.595899 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-envoy" pod "36fb3684-6cf3-4d2d-b930-ad0388420357" (UID: "36fb3684-6cf3-4d2d-b930-ad0388420357"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:13.625993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.620221 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6nq5\" (UniqueName: \"kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-kube-api-access-d6nq5\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.625993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.620254 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.625993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.620268 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36fb3684-6cf3-4d2d-b930-ad0388420357-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.649872 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.649506 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:27:13.743376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.742983 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-envoy\") pod \"7ca77461-d958-4246-8ca8-0c45925a9e6c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " Oct 30 18:27:13.743376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.743048 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-credential-socket\") pod \"7ca77461-d958-4246-8ca8-0c45925a9e6c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " Oct 30 18:27:13.743376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.743091 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-data\") pod \"7ca77461-d958-4246-8ca8-0c45925a9e6c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " Oct 30 18:27:13.743376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.743183 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca77461-d958-4246-8ca8-0c45925a9e6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca77461-d958-4246-8ca8-0c45925a9e6c-istiod-ca-cert\") pod \"7ca77461-d958-4246-8ca8-0c45925a9e6c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " Oct 30 18:27:13.743376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.743222 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-certs\") pod \"7ca77461-d958-4246-8ca8-0c45925a9e6c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " Oct 30 18:27:13.743764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.743430 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-token\" (UniqueName: \"kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-token\") pod \"7ca77461-d958-4246-8ca8-0c45925a9e6c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " Oct 30 18:27:13.743764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.743488 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-podinfo\") pod \"7ca77461-d958-4246-8ca8-0c45925a9e6c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " Oct 30 18:27:13.743764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.743537 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-socket\") pod \"7ca77461-d958-4246-8ca8-0c45925a9e6c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " Oct 30 18:27:13.743764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.743578 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-kube-api-access-sscz4\" (UniqueName: \"kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-kube-api-access-sscz4\") pod \"7ca77461-d958-4246-8ca8-0c45925a9e6c\" (UID: \"7ca77461-d958-4246-8ca8-0c45925a9e6c\") " Oct 30 18:27:13.747988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.745017 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-certs" pod "7ca77461-d958-4246-8ca8-0c45925a9e6c" (UID: "7ca77461-d958-4246-8ca8-0c45925a9e6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:13.748434 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.748385 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-credential-socket" pod "7ca77461-d958-4246-8ca8-0c45925a9e6c" (UID: "7ca77461-d958-4246-8ca8-0c45925a9e6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:13.749227 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.749158 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-data" pod "7ca77461-d958-4246-8ca8-0c45925a9e6c" (UID: "7ca77461-d958-4246-8ca8-0c45925a9e6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:13.762037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.752199 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca77461-d958-4246-8ca8-0c45925a9e6c-istiod-ca-cert" pod "7ca77461-d958-4246-8ca8-0c45925a9e6c" (UID: "7ca77461-d958-4246-8ca8-0c45925a9e6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:13.762574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.762250 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-socket" pod "7ca77461-d958-4246-8ca8-0c45925a9e6c" (UID: "7ca77461-d958-4246-8ca8-0c45925a9e6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:13.766547 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.766520 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-kube-api-access-sscz4" pod "7ca77461-d958-4246-8ca8-0c45925a9e6c" (UID: "7ca77461-d958-4246-8ca8-0c45925a9e6c"). InnerVolumeSpecName "kube-api-access-sscz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:13.788209 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.788143 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-token" pod "7ca77461-d958-4246-8ca8-0c45925a9e6c" (UID: "7ca77461-d958-4246-8ca8-0c45925a9e6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:13.803127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.803062 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-podinfo" pod "7ca77461-d958-4246-8ca8-0c45925a9e6c" (UID: "7ca77461-d958-4246-8ca8-0c45925a9e6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:13.803773 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.803729 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-envoy" pod "7ca77461-d958-4246-8ca8-0c45925a9e6c" (UID: "7ca77461-d958-4246-8ca8-0c45925a9e6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:13.863728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.838687 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:27:13.863728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.839995 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:27:13.863728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.841354 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:27:13.863728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.850134 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.863728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.850163 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.863728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.850178 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.863728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.850197 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sscz4\" (UniqueName: \"kubernetes.io/projected/7ca77461-d958-4246-8ca8-0c45925a9e6c-kube-api-access-sscz4\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.863728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.850211 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.863728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.850227 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.863728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.850240 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.863728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.850254 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca77461-d958-4246-8ca8-0c45925a9e6c-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:13.863728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:13.850266 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca77461-d958-4246-8ca8-0c45925a9e6c-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:14.019280 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.018000 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.048918 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.048857 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:27:14.084424 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.084147 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.589050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.568047 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.589050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.568446 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.642679 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.642638 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:27:14.643226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.643198 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:27:14.643631 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.643609 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:27:14.644075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.644053 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:27:14.712611 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.712526 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:27:14.834783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.834747 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" event={"ID":"36fb3684-6cf3-4d2d-b930-ad0388420357","Type":"ContainerDied","Data":"fdad4cbdd84473a8c40fcd968429828340769c481cb4798c6e601cdfadaa418b"} Oct 30 18:27:14.835074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.835057 4883 scope.go:122] "RemoveContainer" containerID="5dff6df0f2c7eac44e32d21ea545d3ea352421c1a597907131a95cae25324f5d" Oct 30 18:27:14.835395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.835380 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7" Oct 30 18:27:14.852765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.847177 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:14.852765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.847555 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:27:14.891991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.880928 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh"] Oct 30 18:27:14.897246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.897059 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" event={"ID":"7ca77461-d958-4246-8ca8-0c45925a9e6c","Type":"ContainerDied","Data":"667750bc60a70ef847d131efdc42006de2637c177deeebc139438ebd7ff9dfe1"} Oct 30 18:27:14.897579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.897556 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c" Oct 30 18:27:14.903443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.902181 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:27:14.946148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:14.946123 4883 scope.go:122] "RemoveContainer" containerID="b371823883b2993929be1ebe01afe19177e8ed9ca53ce7bbcbec75745036038c" Oct 30 18:27:15.050775 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.050607 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:15.051520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.051487 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:15.092960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.086474 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7"] Oct 30 18:27:15.100235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.095456 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:15.114291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.104125 4883 patch_prober.go:29] interesting pod/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:27:15.114291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.104664 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" podUID="549a0999-9c9c-414a-ab9b-8574a8660dbf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:27:15.475750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.475716 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-86gh7"] Oct 30 18:27:15.481153 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.481124 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:27:15.481416 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.481400 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:27:15.630536 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.630498 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:27:15.724569 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.684363 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:15.724569 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.691601 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:27:15.724569 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.720706 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:15.730032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.729949 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:27:15.765955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.751461 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c"] Oct 30 18:27:15.765955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.764483 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c2b103-openshift-default-696dd44677-zds8c"] Oct 30 18:27:15.856954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.798814 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:27:15.856954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.839423 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:27:15.857213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.857129 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:27:15.890961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.858860 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:27:15.903245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.903204 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:27:15.930800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:15.930766 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" event={"ID":"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d","Type":"ContainerStarted","Data":"c071300b7713df2250384be98274c168c46ca029145ede7be77acab94fb9c2f8"} Oct 30 18:27:16.947675 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:16.947639 4883 generic.go:356] "Generic (PLEG): container finished" podID="32c4e4cf-2711-4319-afa8-acd5ec3a6f4d" containerID="bfba356e00942b0cdfac9b07cae4d0602dfb05169495bdbea4412fba1fad9c77" exitCode=0 Oct 30 18:27:16.948171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:16.947686 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" event={"ID":"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d","Type":"ContainerDied","Data":"bfba356e00942b0cdfac9b07cae4d0602dfb05169495bdbea4412fba1fad9c77"} Oct 30 18:27:17.181188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:17.181154 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36fb3684-6cf3-4d2d-b930-ad0388420357" path="/var/lib/kubelet/pods/36fb3684-6cf3-4d2d-b930-ad0388420357/volumes" Oct 30 18:27:17.181914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:17.181891 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca77461-d958-4246-8ca8-0c45925a9e6c" path="/var/lib/kubelet/pods/7ca77461-d958-4246-8ca8-0c45925a9e6c/volumes" Oct 30 18:27:18.137213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.137176 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:18.247072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.246994 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-openshift-ca\") pod \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\" (UID: \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\") " Oct 30 18:27:18.247072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.247038 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-kube-api-access-9n8pz\" (UniqueName: \"kubernetes.io/projected/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-kube-api-access-9n8pz\") pod \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\" (UID: \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\") " Oct 30 18:27:18.247272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.247175 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-xks-ca\") pod \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\" (UID: \"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d\") " Oct 30 18:27:18.247454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.247431 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-openshift-ca" pod "32c4e4cf-2711-4319-afa8-acd5ec3a6f4d" (UID: "32c4e4cf-2711-4319-afa8-acd5ec3a6f4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:18.247807 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.247781 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:18.249472 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.249445 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-xks-ca" pod "32c4e4cf-2711-4319-afa8-acd5ec3a6f4d" (UID: "32c4e4cf-2711-4319-afa8-acd5ec3a6f4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:18.249582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.249472 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-kube-api-access-9n8pz" pod "32c4e4cf-2711-4319-afa8-acd5ec3a6f4d" (UID: "32c4e4cf-2711-4319-afa8-acd5ec3a6f4d"). InnerVolumeSpecName "kube-api-access-9n8pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:18.349190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.349153 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n8pz\" (UniqueName: \"kubernetes.io/projected/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-kube-api-access-9n8pz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:18.349190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.349186 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:18.968542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.968417 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" Oct 30 18:27:18.968542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.968455 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh" event={"ID":"32c4e4cf-2711-4319-afa8-acd5ec3a6f4d","Type":"ContainerDied","Data":"c071300b7713df2250384be98274c168c46ca029145ede7be77acab94fb9c2f8"} Oct 30 18:27:18.968542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:18.968509 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c071300b7713df2250384be98274c168c46ca029145ede7be77acab94fb9c2f8" Oct 30 18:27:19.977800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:19.977769 4883 generic.go:356] "Generic (PLEG): container finished" podID="549a0999-9c9c-414a-ab9b-8574a8660dbf" containerID="5785b57fb390aec729843b19a0fb7ef4211a37f267eea03f94a143b5d0a723b5" exitCode=0 Oct 30 18:27:19.978244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:19.977842 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" event={"ID":"549a0999-9c9c-414a-ab9b-8574a8660dbf","Type":"ContainerDied","Data":"5785b57fb390aec729843b19a0fb7ef4211a37f267eea03f94a143b5d0a723b5"} Oct 30 18:27:20.115176 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.115150 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:27:20.279927 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.279840 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-credential-socket\") pod \"549a0999-9c9c-414a-ab9b-8574a8660dbf\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " Oct 30 18:27:20.279927 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.279885 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549a0999-9c9c-414a-ab9b-8574a8660dbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549a0999-9c9c-414a-ab9b-8574a8660dbf-istiod-ca-cert\") pod \"549a0999-9c9c-414a-ab9b-8574a8660dbf\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " Oct 30 18:27:20.280181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.279987 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-socket\") pod \"549a0999-9c9c-414a-ab9b-8574a8660dbf\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " Oct 30 18:27:20.280181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.280082 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-data\") pod \"549a0999-9c9c-414a-ab9b-8574a8660dbf\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " Oct 30 18:27:20.280181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.280158 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-podinfo\") pod \"549a0999-9c9c-414a-ab9b-8574a8660dbf\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " Oct 30 18:27:20.280347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.280181 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-credential-socket" pod "549a0999-9c9c-414a-ab9b-8574a8660dbf" (UID: "549a0999-9c9c-414a-ab9b-8574a8660dbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:20.280347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.280265 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-token\" (UniqueName: \"kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-token\") pod \"549a0999-9c9c-414a-ab9b-8574a8660dbf\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " Oct 30 18:27:20.280347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.280334 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-envoy\") pod \"549a0999-9c9c-414a-ab9b-8574a8660dbf\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " Oct 30 18:27:20.280451 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.280337 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-data" pod "549a0999-9c9c-414a-ab9b-8574a8660dbf" (UID: "549a0999-9c9c-414a-ab9b-8574a8660dbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:20.280451 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.280383 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-kube-api-access-d4vm6\" (UniqueName: \"kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-kube-api-access-d4vm6\") pod \"549a0999-9c9c-414a-ab9b-8574a8660dbf\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " Oct 30 18:27:20.280451 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.280389 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-socket" pod "549a0999-9c9c-414a-ab9b-8574a8660dbf" (UID: "549a0999-9c9c-414a-ab9b-8574a8660dbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:20.280451 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.280437 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549a0999-9c9c-414a-ab9b-8574a8660dbf-istiod-ca-cert" pod "549a0999-9c9c-414a-ab9b-8574a8660dbf" (UID: "549a0999-9c9c-414a-ab9b-8574a8660dbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:20.280691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.280471 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-certs\") pod \"549a0999-9c9c-414a-ab9b-8574a8660dbf\" (UID: \"549a0999-9c9c-414a-ab9b-8574a8660dbf\") " Oct 30 18:27:20.280758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.280675 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-certs" pod "549a0999-9c9c-414a-ab9b-8574a8660dbf" (UID: "549a0999-9c9c-414a-ab9b-8574a8660dbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:20.281190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.281164 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:20.281331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.281193 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:20.281331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.281208 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:20.281331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.281222 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549a0999-9c9c-414a-ab9b-8574a8660dbf-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:20.281331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.281236 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:20.282527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.282500 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-envoy" pod "549a0999-9c9c-414a-ab9b-8574a8660dbf" (UID: "549a0999-9c9c-414a-ab9b-8574a8660dbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:20.282640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.282560 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-podinfo" pod "549a0999-9c9c-414a-ab9b-8574a8660dbf" (UID: "549a0999-9c9c-414a-ab9b-8574a8660dbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:20.282640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.282575 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-token" pod "549a0999-9c9c-414a-ab9b-8574a8660dbf" (UID: "549a0999-9c9c-414a-ab9b-8574a8660dbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:20.282640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.282592 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-kube-api-access-d4vm6" pod "549a0999-9c9c-414a-ab9b-8574a8660dbf" (UID: "549a0999-9c9c-414a-ab9b-8574a8660dbf"). InnerVolumeSpecName "kube-api-access-d4vm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:20.382728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.382689 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:20.382728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.382723 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4vm6\" (UniqueName: \"kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-kube-api-access-d4vm6\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:20.382728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.382735 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:20.382997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.382743 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549a0999-9c9c-414a-ab9b-8574a8660dbf-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:20.987573 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.987540 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" Oct 30 18:27:20.987573 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.987567 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks" event={"ID":"549a0999-9c9c-414a-ab9b-8574a8660dbf","Type":"ContainerDied","Data":"e4509e1e30f328f8c93a87acfbaae83c48e6c9e575c5750f821b750a991d15a2"} Oct 30 18:27:20.988078 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:20.987636 4883 scope.go:122] "RemoveContainer" containerID="5785b57fb390aec729843b19a0fb7ef4211a37f267eea03f94a143b5d0a723b5" Oct 30 18:27:21.058034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.057997 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks"] Oct 30 18:27:21.073685 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.073646 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-42fad7-openshift-default-75699fd96f-6sqks"] Oct 30 18:27:21.179898 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.179849 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549a0999-9c9c-414a-ab9b-8574a8660dbf" path="/var/lib/kubelet/pods/549a0999-9c9c-414a-ab9b-8574a8660dbf/volumes" Oct 30 18:27:21.790197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.790166 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:27:21.796775 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.796753 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:27:21.802022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.802004 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:27:21.807989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.807971 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:27:21.875119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875090 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2"] Oct 30 18:27:21.875623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875610 4883 cpu_manager.go:414] "containerMap: removing container" podUID="32c4e4cf-2711-4319-afa8-acd5ec3a6f4d" containerName="revoke-keys" Oct 30 18:27:21.875667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875625 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32c4e4cf-2711-4319-afa8-acd5ec3a6f4d" containerName="revoke-keys" Oct 30 18:27:21.875667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875653 4883 cpu_manager.go:414] "containerMap: removing container" podUID="549a0999-9c9c-414a-ab9b-8574a8660dbf" containerName="istio-proxy" Oct 30 18:27:21.875667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875658 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549a0999-9c9c-414a-ab9b-8574a8660dbf" containerName="istio-proxy" Oct 30 18:27:21.875766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875679 4883 cpu_manager.go:414] "containerMap: removing container" podUID="7ca77461-d958-4246-8ca8-0c45925a9e6c" containerName="istio-proxy" Oct 30 18:27:21.875766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875684 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca77461-d958-4246-8ca8-0c45925a9e6c" containerName="istio-proxy" Oct 30 18:27:21.875766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875691 4883 cpu_manager.go:414] "containerMap: removing container" podUID="36fb3684-6cf3-4d2d-b930-ad0388420357" containerName="istio-proxy" Oct 30 18:27:21.875766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875696 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36fb3684-6cf3-4d2d-b930-ad0388420357" containerName="istio-proxy" Oct 30 18:27:21.875910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875777 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="32c4e4cf-2711-4319-afa8-acd5ec3a6f4d" containerName="revoke-keys" Oct 30 18:27:21.875910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875789 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="549a0999-9c9c-414a-ab9b-8574a8660dbf" containerName="istio-proxy" Oct 30 18:27:21.875910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875799 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca77461-d958-4246-8ca8-0c45925a9e6c" containerName="istio-proxy" Oct 30 18:27:21.875910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.875804 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="36fb3684-6cf3-4d2d-b930-ad0388420357" containerName="istio-proxy" Oct 30 18:27:21.879046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.879026 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:21.895753 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.895726 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:27:21.895901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.895736 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:27:21.895901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.895802 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:21.896146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.896128 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:27:21.899418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:21.899396 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2"] Oct 30 18:27:22.003841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:22.003810 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r92d7\" (UniqueName: \"kubernetes.io/projected/98f06e8d-eab3-445d-98f5-a300fcddb2b0-kube-api-access-r92d7\") pod \"maas-api-revoke-sub-e4f4b644-hf7r2\" (UID: \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:22.004278 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:22.004018 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f4b644-hf7r2\" (UID: \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:22.004278 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:22.004076 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f4b644-hf7r2\" (UID: \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:22.105799 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:22.105768 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r92d7\" (UniqueName: \"kubernetes.io/projected/98f06e8d-eab3-445d-98f5-a300fcddb2b0-kube-api-access-r92d7\") pod \"maas-api-revoke-sub-e4f4b644-hf7r2\" (UID: \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:22.106003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:22.105889 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f4b644-hf7r2\" (UID: \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:22.106003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:22.105968 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f4b644-hf7r2\" (UID: \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:22.106632 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:22.106608 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f4b644-hf7r2\" (UID: \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:22.108295 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:22.108274 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f4b644-hf7r2\" (UID: \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:22.130581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:22.130549 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r92d7\" (UniqueName: \"kubernetes.io/projected/98f06e8d-eab3-445d-98f5-a300fcddb2b0-kube-api-access-r92d7\") pod \"maas-api-revoke-sub-e4f4b644-hf7r2\" (UID: \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:22.192588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:22.192550 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:22.496769 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:22.496739 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2"] Oct 30 18:27:22.498025 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:27:22.497999 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f06e8d_eab3_445d_98f5_a300fcddb2b0.slice/crio-d6643d57946f8eb971008509ed81dd0393d33cc18eaae35884717e501eb7dfb2 WatchSource:0}: Error finding container d6643d57946f8eb971008509ed81dd0393d33cc18eaae35884717e501eb7dfb2: Status 404 returned error can't find the container with id d6643d57946f8eb971008509ed81dd0393d33cc18eaae35884717e501eb7dfb2 Oct 30 18:27:23.007137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:23.007099 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" event={"ID":"98f06e8d-eab3-445d-98f5-a300fcddb2b0","Type":"ContainerStarted","Data":"ef980095d07dc0a46adba53c5825f5554e6fb9454598723822ec19b7d0d2c9ca"} Oct 30 18:27:23.007586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:23.007173 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" event={"ID":"98f06e8d-eab3-445d-98f5-a300fcddb2b0","Type":"ContainerStarted","Data":"d6643d57946f8eb971008509ed81dd0393d33cc18eaae35884717e501eb7dfb2"} Oct 30 18:27:23.050131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:23.050018 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" podStartSLOduration=2.050000223 podStartE2EDuration="2.050000223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:23.049266506 +0000 UTC m=+2102.006270345" watchObservedRunningTime="2026-10-30 18:27:23.050000223 +0000 UTC m=+2102.007004058" Oct 30 18:27:24.018238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:24.018204 4883 generic.go:356] "Generic (PLEG): container finished" podID="98f06e8d-eab3-445d-98f5-a300fcddb2b0" containerID="ef980095d07dc0a46adba53c5825f5554e6fb9454598723822ec19b7d0d2c9ca" exitCode=0 Oct 30 18:27:24.018606 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:24.018292 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" event={"ID":"98f06e8d-eab3-445d-98f5-a300fcddb2b0","Type":"ContainerDied","Data":"ef980095d07dc0a46adba53c5825f5554e6fb9454598723822ec19b7d0d2c9ca"} Oct 30 18:27:25.181869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:25.181842 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:25.256130 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:25.256088 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-xks-ca\") pod \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\" (UID: \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\") " Oct 30 18:27:25.256331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:25.256183 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-openshift-ca\") pod \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\" (UID: \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\") " Oct 30 18:27:25.256455 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:25.256436 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f06e8d-eab3-445d-98f5-a300fcddb2b0-kube-api-access-r92d7\" (UniqueName: \"kubernetes.io/projected/98f06e8d-eab3-445d-98f5-a300fcddb2b0-kube-api-access-r92d7\") pod \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\" (UID: \"98f06e8d-eab3-445d-98f5-a300fcddb2b0\") " Oct 30 18:27:25.256642 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:25.256601 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-openshift-ca" pod "98f06e8d-eab3-445d-98f5-a300fcddb2b0" (UID: "98f06e8d-eab3-445d-98f5-a300fcddb2b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:25.257545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:25.257516 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:25.258980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:25.258950 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-xks-ca" pod "98f06e8d-eab3-445d-98f5-a300fcddb2b0" (UID: "98f06e8d-eab3-445d-98f5-a300fcddb2b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:25.264659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:25.264617 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f06e8d-eab3-445d-98f5-a300fcddb2b0-kube-api-access-r92d7" pod "98f06e8d-eab3-445d-98f5-a300fcddb2b0" (UID: "98f06e8d-eab3-445d-98f5-a300fcddb2b0"). InnerVolumeSpecName "kube-api-access-r92d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:25.358478 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:25.358448 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r92d7\" (UniqueName: \"kubernetes.io/projected/98f06e8d-eab3-445d-98f5-a300fcddb2b0-kube-api-access-r92d7\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:25.358478 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:25.358477 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f06e8d-eab3-445d-98f5-a300fcddb2b0-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:25.361629 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:25.361606 4883 scope.go:122] "RemoveContainer" containerID="b7b4b43beb6e98b4b6140a7edc3ab152e06fb868f5a3e64b7817dfcf9d653e88" Oct 30 18:27:25.373378 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:25.373351 4883 scope.go:122] "RemoveContainer" containerID="61d5abddae738cc7782261f60897e119ea3ebecc701b166dbc5e3ba55ed5d2b6" Oct 30 18:27:26.035541 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:26.035513 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" Oct 30 18:27:26.035727 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:26.035552 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2" event={"ID":"98f06e8d-eab3-445d-98f5-a300fcddb2b0","Type":"ContainerDied","Data":"d6643d57946f8eb971008509ed81dd0393d33cc18eaae35884717e501eb7dfb2"} Oct 30 18:27:26.035727 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:26.035604 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6643d57946f8eb971008509ed81dd0393d33cc18eaae35884717e501eb7dfb2" Oct 30 18:27:27.825327 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.825284 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj"] Oct 30 18:27:27.826136 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.826113 4883 cpu_manager.go:414] "containerMap: removing container" podUID="98f06e8d-eab3-445d-98f5-a300fcddb2b0" containerName="revoke-keys" Oct 30 18:27:27.826136 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.826136 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f06e8d-eab3-445d-98f5-a300fcddb2b0" containerName="revoke-keys" Oct 30 18:27:27.826430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.826305 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f06e8d-eab3-445d-98f5-a300fcddb2b0" containerName="revoke-keys" Oct 30 18:27:27.838651 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.838614 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:27.845758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.845726 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj"] Oct 30 18:27:27.866258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.866224 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:27.877964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.869841 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:27:27.877964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.869850 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:27:27.877964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.869982 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:27:27.923736 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.923707 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf57c\" (UniqueName: \"kubernetes.io/projected/706d9904-f6e8-487b-92e9-ba25aa0bedac-kube-api-access-hf57c\") pod \"maas-api-revoke-sub-c6d9321d-8nmmj\" (UID: \"706d9904-f6e8-487b-92e9-ba25aa0bedac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:27.923919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.923852 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6d9321d-8nmmj\" (UID: \"706d9904-f6e8-487b-92e9-ba25aa0bedac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:27.924014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:27.923950 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6d9321d-8nmmj\" (UID: \"706d9904-f6e8-487b-92e9-ba25aa0bedac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:28.025090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:28.025055 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf57c\" (UniqueName: \"kubernetes.io/projected/706d9904-f6e8-487b-92e9-ba25aa0bedac-kube-api-access-hf57c\") pod \"maas-api-revoke-sub-c6d9321d-8nmmj\" (UID: \"706d9904-f6e8-487b-92e9-ba25aa0bedac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:28.025287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:28.025179 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6d9321d-8nmmj\" (UID: \"706d9904-f6e8-487b-92e9-ba25aa0bedac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:28.025287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:28.025224 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6d9321d-8nmmj\" (UID: \"706d9904-f6e8-487b-92e9-ba25aa0bedac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:28.026103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:28.026075 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6d9321d-8nmmj\" (UID: \"706d9904-f6e8-487b-92e9-ba25aa0bedac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:28.027611 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:28.027587 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6d9321d-8nmmj\" (UID: \"706d9904-f6e8-487b-92e9-ba25aa0bedac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:28.052826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:28.052800 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf57c\" (UniqueName: \"kubernetes.io/projected/706d9904-f6e8-487b-92e9-ba25aa0bedac-kube-api-access-hf57c\") pod \"maas-api-revoke-sub-c6d9321d-8nmmj\" (UID: \"706d9904-f6e8-487b-92e9-ba25aa0bedac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:28.178438 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:28.178403 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:28.492405 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:28.492377 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj"] Oct 30 18:27:28.493547 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:27:28.493521 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706d9904_f6e8_487b_92e9_ba25aa0bedac.slice/crio-7f75474007592a4321458fc139714a6287ce86cb6bb946e5d32e3982050d8909 WatchSource:0}: Error finding container 7f75474007592a4321458fc139714a6287ce86cb6bb946e5d32e3982050d8909: Status 404 returned error can't find the container with id 7f75474007592a4321458fc139714a6287ce86cb6bb946e5d32e3982050d8909 Oct 30 18:27:29.110957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:29.105056 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" event={"ID":"706d9904-f6e8-487b-92e9-ba25aa0bedac","Type":"ContainerStarted","Data":"48f6de5dc8748b1b9cc978f79d28dac27d8eb98e561ad1142878dcf358b3add1"} Oct 30 18:27:29.110957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:29.105135 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" event={"ID":"706d9904-f6e8-487b-92e9-ba25aa0bedac","Type":"ContainerStarted","Data":"7f75474007592a4321458fc139714a6287ce86cb6bb946e5d32e3982050d8909"} Oct 30 18:27:30.114625 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:30.114590 4883 generic.go:356] "Generic (PLEG): container finished" podID="706d9904-f6e8-487b-92e9-ba25aa0bedac" containerID="48f6de5dc8748b1b9cc978f79d28dac27d8eb98e561ad1142878dcf358b3add1" exitCode=0 Oct 30 18:27:30.115069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:30.114678 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" event={"ID":"706d9904-f6e8-487b-92e9-ba25aa0bedac","Type":"ContainerDied","Data":"48f6de5dc8748b1b9cc978f79d28dac27d8eb98e561ad1142878dcf358b3add1"} Oct 30 18:27:31.287996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:31.287969 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:31.376832 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:31.376743 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706d9904-f6e8-487b-92e9-ba25aa0bedac-kube-api-access-hf57c\" (UniqueName: \"kubernetes.io/projected/706d9904-f6e8-487b-92e9-ba25aa0bedac-kube-api-access-hf57c\") pod \"706d9904-f6e8-487b-92e9-ba25aa0bedac\" (UID: \"706d9904-f6e8-487b-92e9-ba25aa0bedac\") " Oct 30 18:27:31.376832 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:31.376813 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-openshift-ca\") pod \"706d9904-f6e8-487b-92e9-ba25aa0bedac\" (UID: \"706d9904-f6e8-487b-92e9-ba25aa0bedac\") " Oct 30 18:27:31.377088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:31.376967 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-xks-ca\") pod \"706d9904-f6e8-487b-92e9-ba25aa0bedac\" (UID: \"706d9904-f6e8-487b-92e9-ba25aa0bedac\") " Oct 30 18:27:31.377198 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:31.377172 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-openshift-ca" pod "706d9904-f6e8-487b-92e9-ba25aa0bedac" (UID: "706d9904-f6e8-487b-92e9-ba25aa0bedac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:31.377655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:31.377628 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:31.379130 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:31.379105 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706d9904-f6e8-487b-92e9-ba25aa0bedac-kube-api-access-hf57c" pod "706d9904-f6e8-487b-92e9-ba25aa0bedac" (UID: "706d9904-f6e8-487b-92e9-ba25aa0bedac"). InnerVolumeSpecName "kube-api-access-hf57c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:31.379229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:31.379179 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-xks-ca" pod "706d9904-f6e8-487b-92e9-ba25aa0bedac" (UID: "706d9904-f6e8-487b-92e9-ba25aa0bedac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:31.479142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:31.479103 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf57c\" (UniqueName: \"kubernetes.io/projected/706d9904-f6e8-487b-92e9-ba25aa0bedac-kube-api-access-hf57c\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:31.479142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:31.479139 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706d9904-f6e8-487b-92e9-ba25aa0bedac-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:32.132265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:32.132128 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" Oct 30 18:27:32.132265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:32.132163 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj" event={"ID":"706d9904-f6e8-487b-92e9-ba25aa0bedac","Type":"ContainerDied","Data":"7f75474007592a4321458fc139714a6287ce86cb6bb946e5d32e3982050d8909"} Oct 30 18:27:32.132265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:32.132253 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f75474007592a4321458fc139714a6287ce86cb6bb946e5d32e3982050d8909" Oct 30 18:27:35.082838 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:27:35.082753 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:27:38.350423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.350389 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s"] Oct 30 18:27:38.351032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.351005 4883 cpu_manager.go:414] "containerMap: removing container" podUID="706d9904-f6e8-487b-92e9-ba25aa0bedac" containerName="revoke-keys" Oct 30 18:27:38.351032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.351024 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706d9904-f6e8-487b-92e9-ba25aa0bedac" containerName="revoke-keys" Oct 30 18:27:38.351408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.351162 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="706d9904-f6e8-487b-92e9-ba25aa0bedac" containerName="revoke-keys" Oct 30 18:27:38.355425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.355403 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:38.387726 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.387674 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:38.388089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.387876 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:27:38.388233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.388090 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:27:38.388233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.388191 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s"] Oct 30 18:27:38.388493 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.388194 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:27:38.481534 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.481493 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-103c5bee-xpb6s\" (UID: \"d21df428-2c21-4e18-b29e-2e7deca7274e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:38.481771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.481744 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-103c5bee-xpb6s\" (UID: \"d21df428-2c21-4e18-b29e-2e7deca7274e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:38.481821 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.481790 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gnwx\" (UniqueName: \"kubernetes.io/projected/d21df428-2c21-4e18-b29e-2e7deca7274e-kube-api-access-2gnwx\") pod \"maas-api-revoke-sub-103c5bee-xpb6s\" (UID: \"d21df428-2c21-4e18-b29e-2e7deca7274e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:38.583071 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.583031 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-103c5bee-xpb6s\" (UID: \"d21df428-2c21-4e18-b29e-2e7deca7274e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:38.583260 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.583090 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gnwx\" (UniqueName: \"kubernetes.io/projected/d21df428-2c21-4e18-b29e-2e7deca7274e-kube-api-access-2gnwx\") pod \"maas-api-revoke-sub-103c5bee-xpb6s\" (UID: \"d21df428-2c21-4e18-b29e-2e7deca7274e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:38.583354 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.583264 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-103c5bee-xpb6s\" (UID: \"d21df428-2c21-4e18-b29e-2e7deca7274e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:38.583900 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.583878 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-103c5bee-xpb6s\" (UID: \"d21df428-2c21-4e18-b29e-2e7deca7274e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:38.585426 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.585404 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-103c5bee-xpb6s\" (UID: \"d21df428-2c21-4e18-b29e-2e7deca7274e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:38.608085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.608019 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gnwx\" (UniqueName: \"kubernetes.io/projected/d21df428-2c21-4e18-b29e-2e7deca7274e-kube-api-access-2gnwx\") pod \"maas-api-revoke-sub-103c5bee-xpb6s\" (UID: \"d21df428-2c21-4e18-b29e-2e7deca7274e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:38.668899 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.668866 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:38.929198 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.929113 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7"] Oct 30 18:27:38.936006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.935978 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:38.958956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:38.958917 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7"] Oct 30 18:27:39.004075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.004032 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s"] Oct 30 18:27:39.118177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.118143 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a8d4878-jqzv7\" (UID: \"891c9dfe-3da4-480d-867d-54295c2fa8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:39.118343 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.118267 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mp64\" (UniqueName: \"kubernetes.io/projected/891c9dfe-3da4-480d-867d-54295c2fa8c6-kube-api-access-2mp64\") pod \"maas-api-revoke-sub-8a8d4878-jqzv7\" (UID: \"891c9dfe-3da4-480d-867d-54295c2fa8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:39.118467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.118453 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a8d4878-jqzv7\" (UID: \"891c9dfe-3da4-480d-867d-54295c2fa8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:39.184929 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.184894 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" event={"ID":"d21df428-2c21-4e18-b29e-2e7deca7274e","Type":"ContainerStarted","Data":"df3af3a36344fb254b1550c3fe6bffbe75ac04e8cca08c20c397869a9bd03c20"} Oct 30 18:27:39.219639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.219608 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a8d4878-jqzv7\" (UID: \"891c9dfe-3da4-480d-867d-54295c2fa8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:39.219781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.219667 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a8d4878-jqzv7\" (UID: \"891c9dfe-3da4-480d-867d-54295c2fa8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:39.219781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.219723 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mp64\" (UniqueName: \"kubernetes.io/projected/891c9dfe-3da4-480d-867d-54295c2fa8c6-kube-api-access-2mp64\") pod \"maas-api-revoke-sub-8a8d4878-jqzv7\" (UID: \"891c9dfe-3da4-480d-867d-54295c2fa8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:39.220253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.220233 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a8d4878-jqzv7\" (UID: \"891c9dfe-3da4-480d-867d-54295c2fa8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:39.222039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.222015 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a8d4878-jqzv7\" (UID: \"891c9dfe-3da4-480d-867d-54295c2fa8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:39.243915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.243880 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mp64\" (UniqueName: \"kubernetes.io/projected/891c9dfe-3da4-480d-867d-54295c2fa8c6-kube-api-access-2mp64\") pod \"maas-api-revoke-sub-8a8d4878-jqzv7\" (UID: \"891c9dfe-3da4-480d-867d-54295c2fa8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:39.254425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.254405 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:39.562243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:39.562218 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7"] Oct 30 18:27:39.563413 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:27:39.563388 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891c9dfe_3da4_480d_867d_54295c2fa8c6.slice/crio-1f960e6f8f00601a639e35e72cebb3d84eaa3ab6decb827912a26ded594869a7 WatchSource:0}: Error finding container 1f960e6f8f00601a639e35e72cebb3d84eaa3ab6decb827912a26ded594869a7: Status 404 returned error can't find the container with id 1f960e6f8f00601a639e35e72cebb3d84eaa3ab6decb827912a26ded594869a7 Oct 30 18:27:40.193665 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:40.193584 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" event={"ID":"891c9dfe-3da4-480d-867d-54295c2fa8c6","Type":"ContainerStarted","Data":"ca6c1f71917204c39e18a90db63c39db0cb067d3b12cd8a6bb3df08fcd740e5a"} Oct 30 18:27:40.193665 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:40.193640 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" event={"ID":"891c9dfe-3da4-480d-867d-54295c2fa8c6","Type":"ContainerStarted","Data":"1f960e6f8f00601a639e35e72cebb3d84eaa3ab6decb827912a26ded594869a7"} Oct 30 18:27:40.195209 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:40.195182 4883 generic.go:356] "Generic (PLEG): container finished" podID="d21df428-2c21-4e18-b29e-2e7deca7274e" containerID="070820b104a0d0d45f99538decb8e9a0da1dcd1a06a5864759da80b0cb7c2d33" exitCode=0 Oct 30 18:27:40.195318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:40.195217 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" event={"ID":"d21df428-2c21-4e18-b29e-2e7deca7274e","Type":"ContainerDied","Data":"070820b104a0d0d45f99538decb8e9a0da1dcd1a06a5864759da80b0cb7c2d33"} Oct 30 18:27:40.238855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:40.238805 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" podStartSLOduration=2.238793093 podStartE2EDuration="2.238793093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:40.235991311 +0000 UTC m=+2119.192995147" watchObservedRunningTime="2026-10-30 18:27:40.238793093 +0000 UTC m=+2119.195796928" Oct 30 18:27:41.209214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:41.209166 4883 generic.go:356] "Generic (PLEG): container finished" podID="891c9dfe-3da4-480d-867d-54295c2fa8c6" containerID="ca6c1f71917204c39e18a90db63c39db0cb067d3b12cd8a6bb3df08fcd740e5a" exitCode=0 Oct 30 18:27:41.209735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:41.209293 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" event={"ID":"891c9dfe-3da4-480d-867d-54295c2fa8c6","Type":"ContainerDied","Data":"ca6c1f71917204c39e18a90db63c39db0cb067d3b12cd8a6bb3df08fcd740e5a"} Oct 30 18:27:41.409082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:41.409052 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:41.451783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:41.451753 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-xks-ca\") pod \"d21df428-2c21-4e18-b29e-2e7deca7274e\" (UID: \"d21df428-2c21-4e18-b29e-2e7deca7274e\") " Oct 30 18:27:41.452005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:41.451827 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21df428-2c21-4e18-b29e-2e7deca7274e-kube-api-access-2gnwx\" (UniqueName: \"kubernetes.io/projected/d21df428-2c21-4e18-b29e-2e7deca7274e-kube-api-access-2gnwx\") pod \"d21df428-2c21-4e18-b29e-2e7deca7274e\" (UID: \"d21df428-2c21-4e18-b29e-2e7deca7274e\") " Oct 30 18:27:41.452005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:41.451894 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-openshift-ca\") pod \"d21df428-2c21-4e18-b29e-2e7deca7274e\" (UID: \"d21df428-2c21-4e18-b29e-2e7deca7274e\") " Oct 30 18:27:41.452311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:41.452283 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-openshift-ca" pod "d21df428-2c21-4e18-b29e-2e7deca7274e" (UID: "d21df428-2c21-4e18-b29e-2e7deca7274e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:41.454157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:41.454131 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-xks-ca" pod "d21df428-2c21-4e18-b29e-2e7deca7274e" (UID: "d21df428-2c21-4e18-b29e-2e7deca7274e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:41.454279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:41.454165 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21df428-2c21-4e18-b29e-2e7deca7274e-kube-api-access-2gnwx" pod "d21df428-2c21-4e18-b29e-2e7deca7274e" (UID: "d21df428-2c21-4e18-b29e-2e7deca7274e"). InnerVolumeSpecName "kube-api-access-2gnwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:41.553855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:41.553822 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:41.553855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:41.553850 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gnwx\" (UniqueName: \"kubernetes.io/projected/d21df428-2c21-4e18-b29e-2e7deca7274e-kube-api-access-2gnwx\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:41.553855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:41.553860 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21df428-2c21-4e18-b29e-2e7deca7274e-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:42.217973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.217920 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" Oct 30 18:27:42.218364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.217925 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s" event={"ID":"d21df428-2c21-4e18-b29e-2e7deca7274e","Type":"ContainerDied","Data":"df3af3a36344fb254b1550c3fe6bffbe75ac04e8cca08c20c397869a9bd03c20"} Oct 30 18:27:42.218364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.218069 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df3af3a36344fb254b1550c3fe6bffbe75ac04e8cca08c20c397869a9bd03c20" Oct 30 18:27:42.379406 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.379370 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:42.464769 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.464741 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891c9dfe-3da4-480d-867d-54295c2fa8c6-kube-api-access-2mp64\" (UniqueName: \"kubernetes.io/projected/891c9dfe-3da4-480d-867d-54295c2fa8c6-kube-api-access-2mp64\") pod \"891c9dfe-3da4-480d-867d-54295c2fa8c6\" (UID: \"891c9dfe-3da4-480d-867d-54295c2fa8c6\") " Oct 30 18:27:42.464989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.464968 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-openshift-ca\") pod \"891c9dfe-3da4-480d-867d-54295c2fa8c6\" (UID: \"891c9dfe-3da4-480d-867d-54295c2fa8c6\") " Oct 30 18:27:42.465073 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.464991 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-xks-ca\") pod \"891c9dfe-3da4-480d-867d-54295c2fa8c6\" (UID: \"891c9dfe-3da4-480d-867d-54295c2fa8c6\") " Oct 30 18:27:42.465325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.465300 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-openshift-ca" pod "891c9dfe-3da4-480d-867d-54295c2fa8c6" (UID: "891c9dfe-3da4-480d-867d-54295c2fa8c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:42.465474 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.465451 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:42.467004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.466980 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-xks-ca" pod "891c9dfe-3da4-480d-867d-54295c2fa8c6" (UID: "891c9dfe-3da4-480d-867d-54295c2fa8c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:42.467075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.467019 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891c9dfe-3da4-480d-867d-54295c2fa8c6-kube-api-access-2mp64" pod "891c9dfe-3da4-480d-867d-54295c2fa8c6" (UID: "891c9dfe-3da4-480d-867d-54295c2fa8c6"). InnerVolumeSpecName "kube-api-access-2mp64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:42.567153 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.567122 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/891c9dfe-3da4-480d-867d-54295c2fa8c6-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:42.567153 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:42.567148 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mp64\" (UniqueName: \"kubernetes.io/projected/891c9dfe-3da4-480d-867d-54295c2fa8c6-kube-api-access-2mp64\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:43.227661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:43.227634 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" Oct 30 18:27:43.228024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:43.227662 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7" event={"ID":"891c9dfe-3da4-480d-867d-54295c2fa8c6","Type":"ContainerDied","Data":"1f960e6f8f00601a639e35e72cebb3d84eaa3ab6decb827912a26ded594869a7"} Oct 30 18:27:43.228024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:43.227709 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f960e6f8f00601a639e35e72cebb3d84eaa3ab6decb827912a26ded594869a7" Oct 30 18:27:50.573892 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.573854 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp"] Oct 30 18:27:50.575833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.575793 4883 cpu_manager.go:414] "containerMap: removing container" podUID="891c9dfe-3da4-480d-867d-54295c2fa8c6" containerName="revoke-keys" Oct 30 18:27:50.576093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.576063 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="891c9dfe-3da4-480d-867d-54295c2fa8c6" containerName="revoke-keys" Oct 30 18:27:50.576318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.576286 4883 cpu_manager.go:414] "containerMap: removing container" podUID="d21df428-2c21-4e18-b29e-2e7deca7274e" containerName="revoke-keys" Oct 30 18:27:50.576318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.576318 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21df428-2c21-4e18-b29e-2e7deca7274e" containerName="revoke-keys" Oct 30 18:27:50.576602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.576586 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="891c9dfe-3da4-480d-867d-54295c2fa8c6" containerName="revoke-keys" Oct 30 18:27:50.576721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.576613 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21df428-2c21-4e18-b29e-2e7deca7274e" containerName="revoke-keys" Oct 30 18:27:50.581334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.581303 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:50.600790 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.600754 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp"] Oct 30 18:27:50.610968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.610921 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:27:50.611266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.611251 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:50.611442 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.611429 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:27:50.611603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.611592 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:27:50.697911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.697876 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f463ca-dkdxp\" (UID: \"0e18119b-d51e-4470-8a2b-285c1bd0b905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:50.697911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.697914 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26b2c\" (UniqueName: \"kubernetes.io/projected/0e18119b-d51e-4470-8a2b-285c1bd0b905-kube-api-access-26b2c\") pod \"maas-api-revoke-sub-79f463ca-dkdxp\" (UID: \"0e18119b-d51e-4470-8a2b-285c1bd0b905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:50.698184 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.697958 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f463ca-dkdxp\" (UID: \"0e18119b-d51e-4470-8a2b-285c1bd0b905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:50.799175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.799143 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f463ca-dkdxp\" (UID: \"0e18119b-d51e-4470-8a2b-285c1bd0b905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:50.799326 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.799187 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26b2c\" (UniqueName: \"kubernetes.io/projected/0e18119b-d51e-4470-8a2b-285c1bd0b905-kube-api-access-26b2c\") pod \"maas-api-revoke-sub-79f463ca-dkdxp\" (UID: \"0e18119b-d51e-4470-8a2b-285c1bd0b905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:50.799326 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.799301 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f463ca-dkdxp\" (UID: \"0e18119b-d51e-4470-8a2b-285c1bd0b905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:50.799818 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.799801 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79f463ca-dkdxp\" (UID: \"0e18119b-d51e-4470-8a2b-285c1bd0b905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:50.801862 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.801839 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79f463ca-dkdxp\" (UID: \"0e18119b-d51e-4470-8a2b-285c1bd0b905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:50.823635 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.823609 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26b2c\" (UniqueName: \"kubernetes.io/projected/0e18119b-d51e-4470-8a2b-285c1bd0b905-kube-api-access-26b2c\") pod \"maas-api-revoke-sub-79f463ca-dkdxp\" (UID: \"0e18119b-d51e-4470-8a2b-285c1bd0b905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:50.916057 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:50.916016 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:51.222983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:51.222950 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp"] Oct 30 18:27:51.223171 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:27:51.223150 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e18119b_d51e_4470_8a2b_285c1bd0b905.slice/crio-ac48c2c65bb300d1c0eb46832ccf7be1522efb970fd09b2cb757551bed9a78dd WatchSource:0}: Error finding container ac48c2c65bb300d1c0eb46832ccf7be1522efb970fd09b2cb757551bed9a78dd: Status 404 returned error can't find the container with id ac48c2c65bb300d1c0eb46832ccf7be1522efb970fd09b2cb757551bed9a78dd Oct 30 18:27:51.291095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:51.291046 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" event={"ID":"0e18119b-d51e-4470-8a2b-285c1bd0b905","Type":"ContainerStarted","Data":"ac48c2c65bb300d1c0eb46832ccf7be1522efb970fd09b2cb757551bed9a78dd"} Oct 30 18:27:52.301275 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.301239 4883 generic.go:356] "Generic (PLEG): container finished" podID="0e18119b-d51e-4470-8a2b-285c1bd0b905" containerID="1587bbb9849e73baa967e8e49181818b0f75dfa64d99b453982c3980715937b1" exitCode=0 Oct 30 18:27:52.301677 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.301321 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" event={"ID":"0e18119b-d51e-4470-8a2b-285c1bd0b905","Type":"ContainerDied","Data":"1587bbb9849e73baa967e8e49181818b0f75dfa64d99b453982c3980715937b1"} Oct 30 18:27:52.615497 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.615457 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q"] Oct 30 18:27:52.619764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.619732 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:52.639118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.639091 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q"] Oct 30 18:27:52.735894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.735857 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b34c5be7-hgz2q\" (UID: \"59f14729-a34e-477b-9a84-4fe5af1175fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:52.736074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.735911 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b34c5be7-hgz2q\" (UID: \"59f14729-a34e-477b-9a84-4fe5af1175fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:52.736137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.736100 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpqwj\" (UniqueName: \"kubernetes.io/projected/59f14729-a34e-477b-9a84-4fe5af1175fa-kube-api-access-vpqwj\") pod \"maas-api-revoke-sub-b34c5be7-hgz2q\" (UID: \"59f14729-a34e-477b-9a84-4fe5af1175fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:52.837561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.837527 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b34c5be7-hgz2q\" (UID: \"59f14729-a34e-477b-9a84-4fe5af1175fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:52.837735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.837582 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b34c5be7-hgz2q\" (UID: \"59f14729-a34e-477b-9a84-4fe5af1175fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:52.837735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.837654 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqwj\" (UniqueName: \"kubernetes.io/projected/59f14729-a34e-477b-9a84-4fe5af1175fa-kube-api-access-vpqwj\") pod \"maas-api-revoke-sub-b34c5be7-hgz2q\" (UID: \"59f14729-a34e-477b-9a84-4fe5af1175fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:52.838296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.838275 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b34c5be7-hgz2q\" (UID: \"59f14729-a34e-477b-9a84-4fe5af1175fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:52.839910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.839891 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b34c5be7-hgz2q\" (UID: \"59f14729-a34e-477b-9a84-4fe5af1175fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:52.862097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.862071 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpqwj\" (UniqueName: \"kubernetes.io/projected/59f14729-a34e-477b-9a84-4fe5af1175fa-kube-api-access-vpqwj\") pod \"maas-api-revoke-sub-b34c5be7-hgz2q\" (UID: \"59f14729-a34e-477b-9a84-4fe5af1175fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:52.941029 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:52.940930 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:53.254790 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:53.254763 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q"] Oct 30 18:27:53.255615 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:27:53.255590 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f14729_a34e_477b_9a84_4fe5af1175fa.slice/crio-960372f706ac07b7145dbbc09223fc64a3f58fe0543ae2cf6633f0b5dc1585a4 WatchSource:0}: Error finding container 960372f706ac07b7145dbbc09223fc64a3f58fe0543ae2cf6633f0b5dc1585a4: Status 404 returned error can't find the container with id 960372f706ac07b7145dbbc09223fc64a3f58fe0543ae2cf6633f0b5dc1585a4 Oct 30 18:27:53.311338 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:53.311305 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" event={"ID":"59f14729-a34e-477b-9a84-4fe5af1175fa","Type":"ContainerStarted","Data":"960372f706ac07b7145dbbc09223fc64a3f58fe0543ae2cf6633f0b5dc1585a4"} Oct 30 18:27:53.454854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:53.454830 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:53.547739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:53.547647 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-xks-ca\") pod \"0e18119b-d51e-4470-8a2b-285c1bd0b905\" (UID: \"0e18119b-d51e-4470-8a2b-285c1bd0b905\") " Oct 30 18:27:53.547921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:53.547737 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e18119b-d51e-4470-8a2b-285c1bd0b905-kube-api-access-26b2c\" (UniqueName: \"kubernetes.io/projected/0e18119b-d51e-4470-8a2b-285c1bd0b905-kube-api-access-26b2c\") pod \"0e18119b-d51e-4470-8a2b-285c1bd0b905\" (UID: \"0e18119b-d51e-4470-8a2b-285c1bd0b905\") " Oct 30 18:27:53.548041 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:53.548021 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-openshift-ca\") pod \"0e18119b-d51e-4470-8a2b-285c1bd0b905\" (UID: \"0e18119b-d51e-4470-8a2b-285c1bd0b905\") " Oct 30 18:27:53.548325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:53.548303 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-openshift-ca" pod "0e18119b-d51e-4470-8a2b-285c1bd0b905" (UID: "0e18119b-d51e-4470-8a2b-285c1bd0b905"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:53.548505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:53.548487 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:53.549963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:53.549929 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-xks-ca" pod "0e18119b-d51e-4470-8a2b-285c1bd0b905" (UID: "0e18119b-d51e-4470-8a2b-285c1bd0b905"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:53.550017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:53.549956 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e18119b-d51e-4470-8a2b-285c1bd0b905-kube-api-access-26b2c" pod "0e18119b-d51e-4470-8a2b-285c1bd0b905" (UID: "0e18119b-d51e-4470-8a2b-285c1bd0b905"). InnerVolumeSpecName "kube-api-access-26b2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:53.649947 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:53.649905 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e18119b-d51e-4470-8a2b-285c1bd0b905-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:53.650102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:53.649962 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26b2c\" (UniqueName: \"kubernetes.io/projected/0e18119b-d51e-4470-8a2b-285c1bd0b905-kube-api-access-26b2c\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:54.321298 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:54.321268 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" Oct 30 18:27:54.321747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:54.321267 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp" event={"ID":"0e18119b-d51e-4470-8a2b-285c1bd0b905","Type":"ContainerDied","Data":"ac48c2c65bb300d1c0eb46832ccf7be1522efb970fd09b2cb757551bed9a78dd"} Oct 30 18:27:54.321747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:54.321428 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac48c2c65bb300d1c0eb46832ccf7be1522efb970fd09b2cb757551bed9a78dd" Oct 30 18:27:54.322929 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:54.322905 4883 generic.go:356] "Generic (PLEG): container finished" podID="59f14729-a34e-477b-9a84-4fe5af1175fa" containerID="536d4c70252da8ce86910ca74ac60c3d62fad5f77f146c3d190af8c503995f3a" exitCode=0 Oct 30 18:27:54.323082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:54.322964 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" event={"ID":"59f14729-a34e-477b-9a84-4fe5af1175fa","Type":"ContainerDied","Data":"536d4c70252da8ce86910ca74ac60c3d62fad5f77f146c3d190af8c503995f3a"} Oct 30 18:27:55.490975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:55.490947 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:55.575229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:55.575200 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f14729-a34e-477b-9a84-4fe5af1175fa-kube-api-access-vpqwj\" (UniqueName: \"kubernetes.io/projected/59f14729-a34e-477b-9a84-4fe5af1175fa-kube-api-access-vpqwj\") pod \"59f14729-a34e-477b-9a84-4fe5af1175fa\" (UID: \"59f14729-a34e-477b-9a84-4fe5af1175fa\") " Oct 30 18:27:55.575403 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:55.575238 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-xks-ca\") pod \"59f14729-a34e-477b-9a84-4fe5af1175fa\" (UID: \"59f14729-a34e-477b-9a84-4fe5af1175fa\") " Oct 30 18:27:55.575403 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:55.575339 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-openshift-ca\") pod \"59f14729-a34e-477b-9a84-4fe5af1175fa\" (UID: \"59f14729-a34e-477b-9a84-4fe5af1175fa\") " Oct 30 18:27:55.575660 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:55.575637 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-openshift-ca" pod "59f14729-a34e-477b-9a84-4fe5af1175fa" (UID: "59f14729-a34e-477b-9a84-4fe5af1175fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:55.575871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:55.575851 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:55.577384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:55.577357 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-xks-ca" pod "59f14729-a34e-477b-9a84-4fe5af1175fa" (UID: "59f14729-a34e-477b-9a84-4fe5af1175fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:55.577471 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:55.577398 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f14729-a34e-477b-9a84-4fe5af1175fa-kube-api-access-vpqwj" pod "59f14729-a34e-477b-9a84-4fe5af1175fa" (UID: "59f14729-a34e-477b-9a84-4fe5af1175fa"). InnerVolumeSpecName "kube-api-access-vpqwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:55.681685 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:55.681595 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpqwj\" (UniqueName: \"kubernetes.io/projected/59f14729-a34e-477b-9a84-4fe5af1175fa-kube-api-access-vpqwj\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:55.681685 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:55.681638 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f14729-a34e-477b-9a84-4fe5af1175fa-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:27:56.339571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:56.339536 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" event={"ID":"59f14729-a34e-477b-9a84-4fe5af1175fa","Type":"ContainerDied","Data":"960372f706ac07b7145dbbc09223fc64a3f58fe0543ae2cf6633f0b5dc1585a4"} Oct 30 18:27:56.339749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:56.339599 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="960372f706ac07b7145dbbc09223fc64a3f58fe0543ae2cf6633f0b5dc1585a4" Oct 30 18:27:56.339749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:56.339545 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q" Oct 30 18:27:59.325603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.325562 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78"] Oct 30 18:27:59.326338 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.326316 4883 cpu_manager.go:414] "containerMap: removing container" podUID="0e18119b-d51e-4470-8a2b-285c1bd0b905" containerName="revoke-keys" Oct 30 18:27:59.326445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.326339 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e18119b-d51e-4470-8a2b-285c1bd0b905" containerName="revoke-keys" Oct 30 18:27:59.326445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.326368 4883 cpu_manager.go:414] "containerMap: removing container" podUID="59f14729-a34e-477b-9a84-4fe5af1175fa" containerName="revoke-keys" Oct 30 18:27:59.326445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.326376 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f14729-a34e-477b-9a84-4fe5af1175fa" containerName="revoke-keys" Oct 30 18:27:59.326603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.326551 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e18119b-d51e-4470-8a2b-285c1bd0b905" containerName="revoke-keys" Oct 30 18:27:59.326603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.326568 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f14729-a34e-477b-9a84-4fe5af1175fa" containerName="revoke-keys" Oct 30 18:27:59.330253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.330231 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:27:59.350567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.350543 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78"] Oct 30 18:27:59.358849 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.358819 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:27:59.358849 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.358836 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:27:59.358849 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.358821 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:59.359407 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.359393 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:27:59.438371 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.438334 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee130c78-qts78\" (UID: \"775f2dca-b4b5-438d-a835-93bef761df1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:27:59.438516 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.438480 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9bj\" (UniqueName: \"kubernetes.io/projected/775f2dca-b4b5-438d-a835-93bef761df1a-kube-api-access-px9bj\") pod \"maas-api-revoke-sub-ee130c78-qts78\" (UID: \"775f2dca-b4b5-438d-a835-93bef761df1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:27:59.438643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.438626 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee130c78-qts78\" (UID: \"775f2dca-b4b5-438d-a835-93bef761df1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:27:59.539786 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.539752 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee130c78-qts78\" (UID: \"775f2dca-b4b5-438d-a835-93bef761df1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:27:59.540077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.539874 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee130c78-qts78\" (UID: \"775f2dca-b4b5-438d-a835-93bef761df1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:27:59.540077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.539977 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9bj\" (UniqueName: \"kubernetes.io/projected/775f2dca-b4b5-438d-a835-93bef761df1a-kube-api-access-px9bj\") pod \"maas-api-revoke-sub-ee130c78-qts78\" (UID: \"775f2dca-b4b5-438d-a835-93bef761df1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:27:59.540668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.540583 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee130c78-qts78\" (UID: \"775f2dca-b4b5-438d-a835-93bef761df1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:27:59.542197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.542175 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee130c78-qts78\" (UID: \"775f2dca-b4b5-438d-a835-93bef761df1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:27:59.565077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.565043 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9bj\" (UniqueName: \"kubernetes.io/projected/775f2dca-b4b5-438d-a835-93bef761df1a-kube-api-access-px9bj\") pod \"maas-api-revoke-sub-ee130c78-qts78\" (UID: \"775f2dca-b4b5-438d-a835-93bef761df1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:27:59.644316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.644281 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:27:59.997169 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:27:59.997134 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78"] Oct 30 18:27:59.998020 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:27:59.997996 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775f2dca_b4b5_438d_a835_93bef761df1a.slice/crio-da338a9e3b433a3ff332223d59fcb69551ca7a2ebbaa62bc6051d949b058e604 WatchSource:0}: Error finding container da338a9e3b433a3ff332223d59fcb69551ca7a2ebbaa62bc6051d949b058e604: Status 404 returned error can't find the container with id da338a9e3b433a3ff332223d59fcb69551ca7a2ebbaa62bc6051d949b058e604 Oct 30 18:28:00.418919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:00.418885 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" event={"ID":"775f2dca-b4b5-438d-a835-93bef761df1a","Type":"ContainerStarted","Data":"246ce382117226ba8eace738b572b7a64b9c8b9d25850279ae787778f7dc96fe"} Oct 30 18:28:00.419416 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:00.418979 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" event={"ID":"775f2dca-b4b5-438d-a835-93bef761df1a","Type":"ContainerStarted","Data":"da338a9e3b433a3ff332223d59fcb69551ca7a2ebbaa62bc6051d949b058e604"} Oct 30 18:28:00.466621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:00.466571 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" podStartSLOduration=1.466558692 podStartE2EDuration="1.466558692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:00.463151328 +0000 UTC m=+2139.420155165" watchObservedRunningTime="2026-10-30 18:28:00.466558692 +0000 UTC m=+2139.423562527" Oct 30 18:28:01.452958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:01.451620 4883 generic.go:356] "Generic (PLEG): container finished" podID="775f2dca-b4b5-438d-a835-93bef761df1a" containerID="246ce382117226ba8eace738b572b7a64b9c8b9d25850279ae787778f7dc96fe" exitCode=0 Oct 30 18:28:01.452958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:01.451763 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" event={"ID":"775f2dca-b4b5-438d-a835-93bef761df1a","Type":"ContainerDied","Data":"246ce382117226ba8eace738b572b7a64b9c8b9d25850279ae787778f7dc96fe"} Oct 30 18:28:02.638453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:02.638420 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:28:02.694339 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:02.694305 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775f2dca-b4b5-438d-a835-93bef761df1a-kube-api-access-px9bj\" (UniqueName: \"kubernetes.io/projected/775f2dca-b4b5-438d-a835-93bef761df1a-kube-api-access-px9bj\") pod \"775f2dca-b4b5-438d-a835-93bef761df1a\" (UID: \"775f2dca-b4b5-438d-a835-93bef761df1a\") " Oct 30 18:28:02.694525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:02.694399 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-openshift-ca\") pod \"775f2dca-b4b5-438d-a835-93bef761df1a\" (UID: \"775f2dca-b4b5-438d-a835-93bef761df1a\") " Oct 30 18:28:02.694525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:02.694472 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-xks-ca\") pod \"775f2dca-b4b5-438d-a835-93bef761df1a\" (UID: \"775f2dca-b4b5-438d-a835-93bef761df1a\") " Oct 30 18:28:02.695090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:02.695065 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-openshift-ca" pod "775f2dca-b4b5-438d-a835-93bef761df1a" (UID: "775f2dca-b4b5-438d-a835-93bef761df1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:02.696976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:02.696929 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775f2dca-b4b5-438d-a835-93bef761df1a-kube-api-access-px9bj" pod "775f2dca-b4b5-438d-a835-93bef761df1a" (UID: "775f2dca-b4b5-438d-a835-93bef761df1a"). InnerVolumeSpecName "kube-api-access-px9bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:02.697899 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:02.697876 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-xks-ca" pod "775f2dca-b4b5-438d-a835-93bef761df1a" (UID: "775f2dca-b4b5-438d-a835-93bef761df1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:02.796550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:02.796467 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px9bj\" (UniqueName: \"kubernetes.io/projected/775f2dca-b4b5-438d-a835-93bef761df1a-kube-api-access-px9bj\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:02.796550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:02.796496 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:02.796550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:02.796508 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775f2dca-b4b5-438d-a835-93bef761df1a-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:03.469850 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:03.469773 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" Oct 30 18:28:03.470009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:03.469777 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78" event={"ID":"775f2dca-b4b5-438d-a835-93bef761df1a","Type":"ContainerDied","Data":"da338a9e3b433a3ff332223d59fcb69551ca7a2ebbaa62bc6051d949b058e604"} Oct 30 18:28:03.470009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:03.469898 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da338a9e3b433a3ff332223d59fcb69551ca7a2ebbaa62bc6051d949b058e604" Oct 30 18:28:05.081775 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:28:05.081732 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:28:14.970814 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:14.970778 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7"] Oct 30 18:28:14.971537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:14.971517 4883 cpu_manager.go:414] "containerMap: removing container" podUID="775f2dca-b4b5-438d-a835-93bef761df1a" containerName="revoke-keys" Oct 30 18:28:14.971583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:14.971542 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775f2dca-b4b5-438d-a835-93bef761df1a" containerName="revoke-keys" Oct 30 18:28:14.971727 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:14.971715 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="775f2dca-b4b5-438d-a835-93bef761df1a" containerName="revoke-keys" Oct 30 18:28:14.975492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:14.975472 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:14.996342 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:14.996315 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7"] Oct 30 18:28:15.003612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.003573 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:15.003740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.003642 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:28:15.023293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.023263 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:28:15.023293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.023293 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:28:15.076257 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.076221 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-263d960f-bnrx7\" (UID: \"143c1190-bf70-4c79-918f-1d4c891c35db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:15.076430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.076279 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pcwd\" (UniqueName: \"kubernetes.io/projected/143c1190-bf70-4c79-918f-1d4c891c35db-kube-api-access-5pcwd\") pod \"maas-api-revoke-sub-263d960f-bnrx7\" (UID: \"143c1190-bf70-4c79-918f-1d4c891c35db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:15.076474 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.076443 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-263d960f-bnrx7\" (UID: \"143c1190-bf70-4c79-918f-1d4c891c35db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:15.179208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.179165 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-263d960f-bnrx7\" (UID: \"143c1190-bf70-4c79-918f-1d4c891c35db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:15.179406 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.179235 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pcwd\" (UniqueName: \"kubernetes.io/projected/143c1190-bf70-4c79-918f-1d4c891c35db-kube-api-access-5pcwd\") pod \"maas-api-revoke-sub-263d960f-bnrx7\" (UID: \"143c1190-bf70-4c79-918f-1d4c891c35db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:15.179406 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.179386 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-263d960f-bnrx7\" (UID: \"143c1190-bf70-4c79-918f-1d4c891c35db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:15.182799 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.182766 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-263d960f-bnrx7\" (UID: \"143c1190-bf70-4c79-918f-1d4c891c35db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:15.189219 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.189192 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-263d960f-bnrx7\" (UID: \"143c1190-bf70-4c79-918f-1d4c891c35db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:15.206289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.206264 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pcwd\" (UniqueName: \"kubernetes.io/projected/143c1190-bf70-4c79-918f-1d4c891c35db-kube-api-access-5pcwd\") pod \"maas-api-revoke-sub-263d960f-bnrx7\" (UID: \"143c1190-bf70-4c79-918f-1d4c891c35db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:15.289439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.289351 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:15.596625 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:15.596597 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7"] Oct 30 18:28:15.597551 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:28:15.597526 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143c1190_bf70_4c79_918f_1d4c891c35db.slice/crio-06cc9f548e1a7103687fc5dc84fe9a4f6f0d0c066ca37497d2f21269b7ed0338 WatchSource:0}: Error finding container 06cc9f548e1a7103687fc5dc84fe9a4f6f0d0c066ca37497d2f21269b7ed0338: Status 404 returned error can't find the container with id 06cc9f548e1a7103687fc5dc84fe9a4f6f0d0c066ca37497d2f21269b7ed0338 Oct 30 18:28:16.590759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:16.590723 4883 generic.go:356] "Generic (PLEG): container finished" podID="143c1190-bf70-4c79-918f-1d4c891c35db" containerID="9c16dd9d0fbd6ff878fe109598c7310884142171de417609c9970034c8d97c47" exitCode=0 Oct 30 18:28:16.591163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:16.590809 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" event={"ID":"143c1190-bf70-4c79-918f-1d4c891c35db","Type":"ContainerDied","Data":"9c16dd9d0fbd6ff878fe109598c7310884142171de417609c9970034c8d97c47"} Oct 30 18:28:16.591163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:16.590857 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" event={"ID":"143c1190-bf70-4c79-918f-1d4c891c35db","Type":"ContainerStarted","Data":"06cc9f548e1a7103687fc5dc84fe9a4f6f0d0c066ca37497d2f21269b7ed0338"} Oct 30 18:28:17.810209 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:17.810185 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:17.826025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:17.825996 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143c1190-bf70-4c79-918f-1d4c891c35db-kube-api-access-5pcwd\" (UniqueName: \"kubernetes.io/projected/143c1190-bf70-4c79-918f-1d4c891c35db-kube-api-access-5pcwd\") pod \"143c1190-bf70-4c79-918f-1d4c891c35db\" (UID: \"143c1190-bf70-4c79-918f-1d4c891c35db\") " Oct 30 18:28:17.826203 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:17.826135 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-xks-ca\") pod \"143c1190-bf70-4c79-918f-1d4c891c35db\" (UID: \"143c1190-bf70-4c79-918f-1d4c891c35db\") " Oct 30 18:28:17.826279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:17.826261 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-openshift-ca\") pod \"143c1190-bf70-4c79-918f-1d4c891c35db\" (UID: \"143c1190-bf70-4c79-918f-1d4c891c35db\") " Oct 30 18:28:17.826647 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:17.826614 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-openshift-ca" pod "143c1190-bf70-4c79-918f-1d4c891c35db" (UID: "143c1190-bf70-4c79-918f-1d4c891c35db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:17.826863 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:17.826845 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:17.828242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:17.828221 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-xks-ca" pod "143c1190-bf70-4c79-918f-1d4c891c35db" (UID: "143c1190-bf70-4c79-918f-1d4c891c35db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:17.828337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:17.828314 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143c1190-bf70-4c79-918f-1d4c891c35db-kube-api-access-5pcwd" pod "143c1190-bf70-4c79-918f-1d4c891c35db" (UID: "143c1190-bf70-4c79-918f-1d4c891c35db"). InnerVolumeSpecName "kube-api-access-5pcwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:17.928757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:17.928674 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pcwd\" (UniqueName: \"kubernetes.io/projected/143c1190-bf70-4c79-918f-1d4c891c35db-kube-api-access-5pcwd\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:17.928757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:17.928708 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143c1190-bf70-4c79-918f-1d4c891c35db-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:18.608998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:18.608961 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" Oct 30 18:28:18.608998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:18.608971 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7" event={"ID":"143c1190-bf70-4c79-918f-1d4c891c35db","Type":"ContainerDied","Data":"06cc9f548e1a7103687fc5dc84fe9a4f6f0d0c066ca37497d2f21269b7ed0338"} Oct 30 18:28:18.609218 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:18.609027 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06cc9f548e1a7103687fc5dc84fe9a4f6f0d0c066ca37497d2f21269b7ed0338" Oct 30 18:28:22.893678 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:22.893635 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs"] Oct 30 18:28:22.895237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:22.895208 4883 cpu_manager.go:414] "containerMap: removing container" podUID="143c1190-bf70-4c79-918f-1d4c891c35db" containerName="revoke-keys" Oct 30 18:28:22.895388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:22.895371 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143c1190-bf70-4c79-918f-1d4c891c35db" containerName="revoke-keys" Oct 30 18:28:22.895648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:22.895629 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="143c1190-bf70-4c79-918f-1d4c891c35db" containerName="revoke-keys" Oct 30 18:28:22.900395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:22.900372 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:22.926784 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:22.926722 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs"] Oct 30 18:28:22.933124 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:22.932141 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:22.933124 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:22.932267 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:28:22.933124 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:22.932362 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:28:22.933124 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:22.933023 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:28:23.024460 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:23.024353 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3691a87-mnzgs\" (UID: \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:23.024460 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:23.024418 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k54z\" (UniqueName: \"kubernetes.io/projected/fee1c9c5-22ce-49b6-9db6-61431873f1ab-kube-api-access-4k54z\") pod \"maas-api-revoke-sub-d3691a87-mnzgs\" (UID: \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:23.024725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:23.024555 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3691a87-mnzgs\" (UID: \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:23.127359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:23.125712 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3691a87-mnzgs\" (UID: \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:23.127359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:23.125996 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3691a87-mnzgs\" (UID: \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:23.127359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:23.126046 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k54z\" (UniqueName: \"kubernetes.io/projected/fee1c9c5-22ce-49b6-9db6-61431873f1ab-kube-api-access-4k54z\") pod \"maas-api-revoke-sub-d3691a87-mnzgs\" (UID: \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:23.128346 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:23.128319 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3691a87-mnzgs\" (UID: \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:23.145081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:23.136560 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3691a87-mnzgs\" (UID: \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:23.164342 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:23.164267 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k54z\" (UniqueName: \"kubernetes.io/projected/fee1c9c5-22ce-49b6-9db6-61431873f1ab-kube-api-access-4k54z\") pod \"maas-api-revoke-sub-d3691a87-mnzgs\" (UID: \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:23.236167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:23.236133 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:24.432710 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:24.428609 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:24.432710 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:24.429870 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:24.489613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:24.489577 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs"] Oct 30 18:28:24.774268 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:24.773052 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.023583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.023487 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" event={"ID":"fee1c9c5-22ce-49b6-9db6-61431873f1ab","Type":"ContainerStarted","Data":"d63a7840bc17ffa38bf69e02fdc0c8cd008e2821b6823ed64fc8d6ae4c4e528d"} Oct 30 18:28:25.069174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.069130 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.130042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.130002 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.395921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.394058 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.395921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.394506 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:28:25.433330 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.430027 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.433330 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.430700 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:28:25.433330 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.431912 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.433330 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.432523 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:28:25.448339 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.440845 4883 scope.go:122] "RemoveContainer" containerID="f9801000bdc104610306a9af1cead8766dac7d1736f30d196413925fe2cf9238" Oct 30 18:28:25.496422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.495842 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.532353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.532308 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:28:25.643069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.643013 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded" Oct 30 18:28:25.789268 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.789235 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:28:25.792336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.791262 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:28:25.811499 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.789170 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:25.861184 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.861148 4883 scope.go:122] "RemoveContainer" containerID="956415fdd3ad2c301c9043100b1d66f3d9f00635d64dc3de2d17994aa49dd930" Oct 30 18:28:25.990953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:25.990905 4883 scope.go:122] "RemoveContainer" containerID="d91d81f6af607304345bf34ccff0d7ad46973fcd60496dc9e808fb234a3503d2" Oct 30 18:28:26.008121 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.008067 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.073968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.072134 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:28:26.073968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.072706 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:28:26.154140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.151765 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:28:26.154140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.152929 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:28:26.427181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.423451 4883 scope.go:122] "RemoveContainer" containerID="4b383ca609316ee7cb9eeb4e1385e14f0522af367f5e8d61b7283fa646a3dfdc" Oct 30 18:28:26.427181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.423662 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.427181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.424201 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:28:26.427181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.424706 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:28:26.647792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.640444 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:28:26.647792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.643340 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:28:26.665958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.651539 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:28:26.665958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.660312 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:28:26.665958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.660583 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:28:26.745024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.729201 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.745024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.731311 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.808312 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.807650 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.926323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.926272 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.955847 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.955801 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:26.959011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:26.958979 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:28:27.011040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.010967 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:28:27.237191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.128422 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:28:27.351960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.298173 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:28:27.434960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.426152 4883 scope.go:122] "RemoveContainer" containerID="47cb41506d1b2c202811080aa922963f6fbc681126874707534d941ec863e0b9" Oct 30 18:28:27.439087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.439050 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:28:27.574105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.547562 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.574105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.549581 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.617280 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.617231 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.649843 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.649130 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.666534 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.650438 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:28:27.666534 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.658471 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.827722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.827671 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:28:27.832038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.831998 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" event={"ID":"fee1c9c5-22ce-49b6-9db6-61431873f1ab","Type":"ContainerStarted","Data":"44b15cc160dfaf8b764493ef9c2a5199f8292d04497ca1e0c9274200c7838310"} Oct 30 18:28:27.974792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.974689 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.975325 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.975293 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.975668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.975643 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:27.976348 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:27.976328 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:28:28.129972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.129211 4883 scope.go:122] "RemoveContainer" containerID="9ecf6fc546cf10e08068a321910d72030e83a26db4f9ba0fdbb126fd73f0183c" Oct 30 18:28:28.537000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.524155 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.537000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.524430 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.538033 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.537971 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:28:28.538614 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.537978 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:28:28.560989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.550057 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.560989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.550986 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.567639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.567597 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:28:28.568030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.567917 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.592120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.588116 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.626232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.626094 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.655217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.649151 4883 scope.go:122] "RemoveContainer" containerID="6ed0d21db53aa001007609d78cd9e9f6d6f4a1185cec948ca84663a7ee16fd87" Oct 30 18:28:28.655217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.650507 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.655217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.652076 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.663894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.663025 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:28:28.874156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.873708 4883 generic.go:356] "Generic (PLEG): container finished" podID="fee1c9c5-22ce-49b6-9db6-61431873f1ab" containerID="44b15cc160dfaf8b764493ef9c2a5199f8292d04497ca1e0c9274200c7838310" exitCode=0 Oct 30 18:28:28.874156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.873843 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" event={"ID":"fee1c9c5-22ce-49b6-9db6-61431873f1ab","Type":"ContainerDied","Data":"44b15cc160dfaf8b764493ef9c2a5199f8292d04497ca1e0c9274200c7838310"} Oct 30 18:28:28.998009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.994032 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:28.998009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.994435 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:28:28.998009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:28.995431 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:28:29.042687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.042029 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Oct 30 18:28:29.230352 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.230080 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.347303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.347038 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.533045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.532041 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.552206 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.550987 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.552206 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.551313 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:28:29.552206 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.551802 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.553722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.550922 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.626717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.626657 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.667747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.664052 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.667747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.664467 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:28:29.667747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.665860 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:29.792483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:29.791038 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.104020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.011104 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:28:30.104020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.011609 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.104020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.019011 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.104020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.049237 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Oct 30 18:28:30.273979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.246378 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g"] Oct 30 18:28:30.413459 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.397900 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:30.414624 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.414588 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g"] Oct 30 18:28:30.432001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.415825 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0079de08-730b-4941-b639-394b463d5d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebe9ff29-44p4g\" (UID: \"0079de08-730b-4941-b639-394b463d5d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:30.432316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.432287 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrhn\" (UniqueName: \"kubernetes.io/projected/0079de08-730b-4941-b639-394b463d5d3a-kube-api-access-hnrhn\") pod \"maas-api-revoke-sub-ebe9ff29-44p4g\" (UID: \"0079de08-730b-4941-b639-394b463d5d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:30.432702 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.432680 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0079de08-730b-4941-b639-394b463d5d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebe9ff29-44p4g\" (UID: \"0079de08-730b-4941-b639-394b463d5d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:30.574657 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.555207 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.574657 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.573784 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:28:30.585993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.575125 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.585993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.576171 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 18:28:30.585993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.582019 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0079de08-730b-4941-b639-394b463d5d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebe9ff29-44p4g\" (UID: \"0079de08-730b-4941-b639-394b463d5d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:30.585993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.582111 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrhn\" (UniqueName: \"kubernetes.io/projected/0079de08-730b-4941-b639-394b463d5d3a-kube-api-access-hnrhn\") pod \"maas-api-revoke-sub-ebe9ff29-44p4g\" (UID: \"0079de08-730b-4941-b639-394b463d5d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:30.585993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.582313 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0079de08-730b-4941-b639-394b463d5d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebe9ff29-44p4g\" (UID: \"0079de08-730b-4941-b639-394b463d5d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:30.587602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.586498 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:28:30.595975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.590716 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.595975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.595632 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:28:30.595975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.595678 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.596278 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.596042 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.596566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.596496 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0079de08-730b-4941-b639-394b463d5d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebe9ff29-44p4g\" (UID: \"0079de08-730b-4941-b639-394b463d5d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:30.639481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.638154 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.656814 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.656748 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrhn\" (UniqueName: \"kubernetes.io/projected/0079de08-730b-4941-b639-394b463d5d3a-kube-api-access-hnrhn\") pod \"maas-api-revoke-sub-ebe9ff29-44p4g\" (UID: \"0079de08-730b-4941-b639-394b463d5d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:30.686097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.679087 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:30.708016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.688912 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:28:30.708016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.689219 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:28:30.803438 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.803391 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0079de08-730b-4941-b639-394b463d5d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebe9ff29-44p4g\" (UID: \"0079de08-730b-4941-b639-394b463d5d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:30.811553 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.811166 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:30.902979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.901550 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-openshift-ca\") pod \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\" (UID: \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\") " Oct 30 18:28:30.902979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.901740 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-xks-ca\") pod \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\" (UID: \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\") " Oct 30 18:28:30.902979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.901782 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee1c9c5-22ce-49b6-9db6-61431873f1ab-kube-api-access-4k54z\" (UniqueName: \"kubernetes.io/projected/fee1c9c5-22ce-49b6-9db6-61431873f1ab-kube-api-access-4k54z\") pod \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\" (UID: \"fee1c9c5-22ce-49b6-9db6-61431873f1ab\") " Oct 30 18:28:30.914969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.903783 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-openshift-ca" pod "fee1c9c5-22ce-49b6-9db6-61431873f1ab" (UID: "fee1c9c5-22ce-49b6-9db6-61431873f1ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:30.940728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.940609 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee1c9c5-22ce-49b6-9db6-61431873f1ab-kube-api-access-4k54z" pod "fee1c9c5-22ce-49b6-9db6-61431873f1ab" (UID: "fee1c9c5-22ce-49b6-9db6-61431873f1ab"). InnerVolumeSpecName "kube-api-access-4k54z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:30.941417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:30.941375 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-xks-ca" pod "fee1c9c5-22ce-49b6-9db6-61431873f1ab" (UID: "fee1c9c5-22ce-49b6-9db6-61431873f1ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:31.012953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.006130 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:31.012953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.006173 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee1c9c5-22ce-49b6-9db6-61431873f1ab-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:31.012953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.006193 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k54z\" (UniqueName: \"kubernetes.io/projected/fee1c9c5-22ce-49b6-9db6-61431873f1ab-kube-api-access-4k54z\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:31.020344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.019346 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:31.037958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.027047 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:31.037958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.028506 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:31.053239 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.053202 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:31.084989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.075182 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:28:31.137971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.130587 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:28:31.165668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.165637 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:28:31.333567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.333535 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" Oct 30 18:28:31.344164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.344122 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs" event={"ID":"fee1c9c5-22ce-49b6-9db6-61431873f1ab","Type":"ContainerDied","Data":"d63a7840bc17ffa38bf69e02fdc0c8cd008e2821b6823ed64fc8d6ae4c4e528d"} Oct 30 18:28:31.344357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.344173 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d63a7840bc17ffa38bf69e02fdc0c8cd008e2821b6823ed64fc8d6ae4c4e528d" Oct 30 18:28:31.344357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.344209 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:28:31.344357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.344233 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:28:31.344357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.344254 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:28:31.344357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.344274 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:28:31.344357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.344296 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:28:31.344357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.344317 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:28:31.344357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.344337 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:28:31.344357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.344357 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:28:31.344818 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.344377 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:28:31.413973 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:28:31.413914 4883 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee1c9c5_22ce_49b6_9db6_61431873f1ab.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:28:31.607224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:31.607197 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g"] Oct 30 18:28:31.611242 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:28:31.611210 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0079de08_730b_4941_b639_394b463d5d3a.slice/crio-bb0e7e5bf77a6dcd680c1f27e7752de6b606696265b489d08fc9feb34abcf5c9 WatchSource:0}: Error finding container bb0e7e5bf77a6dcd680c1f27e7752de6b606696265b489d08fc9feb34abcf5c9: Status 404 returned error can't find the container with id bb0e7e5bf77a6dcd680c1f27e7752de6b606696265b489d08fc9feb34abcf5c9 Oct 30 18:28:32.343510 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:32.343475 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" event={"ID":"0079de08-730b-4941-b639-394b463d5d3a","Type":"ContainerStarted","Data":"54f77417a44fc6263ca37ffa7715f3c1e0d387b5a1a02fa9813987a4bf2de54d"} Oct 30 18:28:32.343912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:32.343537 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" event={"ID":"0079de08-730b-4941-b639-394b463d5d3a","Type":"ContainerStarted","Data":"bb0e7e5bf77a6dcd680c1f27e7752de6b606696265b489d08fc9feb34abcf5c9"} Oct 30 18:28:32.387729 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:32.387680 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" podStartSLOduration=3.38766631 podStartE2EDuration="3.38766631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:32.385794874 +0000 UTC m=+2171.342798707" watchObservedRunningTime="2026-10-30 18:28:32.38766631 +0000 UTC m=+2171.344670149" Oct 30 18:28:33.352604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:33.352549 4883 generic.go:356] "Generic (PLEG): container finished" podID="0079de08-730b-4941-b639-394b463d5d3a" containerID="54f77417a44fc6263ca37ffa7715f3c1e0d387b5a1a02fa9813987a4bf2de54d" exitCode=0 Oct 30 18:28:33.353362 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:33.352761 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" event={"ID":"0079de08-730b-4941-b639-394b463d5d3a","Type":"ContainerDied","Data":"54f77417a44fc6263ca37ffa7715f3c1e0d387b5a1a02fa9813987a4bf2de54d"} Oct 30 18:28:34.518842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:34.518814 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:34.612370 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:34.612268 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0079de08-730b-4941-b639-394b463d5d3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0079de08-730b-4941-b639-394b463d5d3a-maas-api-openshift-ca\") pod \"0079de08-730b-4941-b639-394b463d5d3a\" (UID: \"0079de08-730b-4941-b639-394b463d5d3a\") " Oct 30 18:28:34.612616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:34.612557 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0079de08-730b-4941-b639-394b463d5d3a-kube-api-access-hnrhn\" (UniqueName: \"kubernetes.io/projected/0079de08-730b-4941-b639-394b463d5d3a-kube-api-access-hnrhn\") pod \"0079de08-730b-4941-b639-394b463d5d3a\" (UID: \"0079de08-730b-4941-b639-394b463d5d3a\") " Oct 30 18:28:34.612789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:34.612768 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0079de08-730b-4941-b639-394b463d5d3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0079de08-730b-4941-b639-394b463d5d3a-maas-api-xks-ca\") pod \"0079de08-730b-4941-b639-394b463d5d3a\" (UID: \"0079de08-730b-4941-b639-394b463d5d3a\") " Oct 30 18:28:34.613492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:34.613465 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0079de08-730b-4941-b639-394b463d5d3a-maas-api-openshift-ca" pod "0079de08-730b-4941-b639-394b463d5d3a" (UID: "0079de08-730b-4941-b639-394b463d5d3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:34.616521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:34.616472 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0079de08-730b-4941-b639-394b463d5d3a-maas-api-xks-ca" pod "0079de08-730b-4941-b639-394b463d5d3a" (UID: "0079de08-730b-4941-b639-394b463d5d3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:34.632166 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:34.632128 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0079de08-730b-4941-b639-394b463d5d3a-kube-api-access-hnrhn" pod "0079de08-730b-4941-b639-394b463d5d3a" (UID: "0079de08-730b-4941-b639-394b463d5d3a"). InnerVolumeSpecName "kube-api-access-hnrhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:34.715202 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:34.715166 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0079de08-730b-4941-b639-394b463d5d3a-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:34.715202 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:34.715196 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnrhn\" (UniqueName: \"kubernetes.io/projected/0079de08-730b-4941-b639-394b463d5d3a-kube-api-access-hnrhn\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:34.715202 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:34.715210 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0079de08-730b-4941-b639-394b463d5d3a-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:35.080525 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:28:35.080435 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:28:35.370772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:35.370741 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" Oct 30 18:28:35.371010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:35.370788 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g" event={"ID":"0079de08-730b-4941-b639-394b463d5d3a","Type":"ContainerDied","Data":"bb0e7e5bf77a6dcd680c1f27e7752de6b606696265b489d08fc9feb34abcf5c9"} Oct 30 18:28:35.371010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:35.370839 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb0e7e5bf77a6dcd680c1f27e7752de6b606696265b489d08fc9feb34abcf5c9" Oct 30 18:28:36.132783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.132744 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg"] Oct 30 18:28:36.133363 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.133344 4883 cpu_manager.go:414] "containerMap: removing container" podUID="0079de08-730b-4941-b639-394b463d5d3a" containerName="revoke-keys" Oct 30 18:28:36.133409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.133364 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0079de08-730b-4941-b639-394b463d5d3a" containerName="revoke-keys" Oct 30 18:28:36.133409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.133392 4883 cpu_manager.go:414] "containerMap: removing container" podUID="fee1c9c5-22ce-49b6-9db6-61431873f1ab" containerName="revoke-keys" Oct 30 18:28:36.133409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.133398 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee1c9c5-22ce-49b6-9db6-61431873f1ab" containerName="revoke-keys" Oct 30 18:28:36.133527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.133517 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="0079de08-730b-4941-b639-394b463d5d3a" containerName="revoke-keys" Oct 30 18:28:36.133563 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.133533 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee1c9c5-22ce-49b6-9db6-61431873f1ab" containerName="revoke-keys" Oct 30 18:28:36.136814 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.136771 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:36.153797 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.153772 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:36.166884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.166854 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:28:36.166884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.166870 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:28:36.167139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.167060 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:28:36.171417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.171388 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg"] Oct 30 18:28:36.337515 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.337482 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d636b736-wjrmg\" (UID: \"2ed55dba-0cb9-45c9-970e-6607ba087a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:36.337699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.337605 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d636b736-wjrmg\" (UID: \"2ed55dba-0cb9-45c9-970e-6607ba087a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:36.337699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.337668 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfhjc\" (UniqueName: \"kubernetes.io/projected/2ed55dba-0cb9-45c9-970e-6607ba087a21-kube-api-access-xfhjc\") pod \"maas-api-revoke-sub-d636b736-wjrmg\" (UID: \"2ed55dba-0cb9-45c9-970e-6607ba087a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:36.439255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.439157 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d636b736-wjrmg\" (UID: \"2ed55dba-0cb9-45c9-970e-6607ba087a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:36.439519 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.439468 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfhjc\" (UniqueName: \"kubernetes.io/projected/2ed55dba-0cb9-45c9-970e-6607ba087a21-kube-api-access-xfhjc\") pod \"maas-api-revoke-sub-d636b736-wjrmg\" (UID: \"2ed55dba-0cb9-45c9-970e-6607ba087a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:36.439900 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.439875 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d636b736-wjrmg\" (UID: \"2ed55dba-0cb9-45c9-970e-6607ba087a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:36.440031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.439907 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d636b736-wjrmg\" (UID: \"2ed55dba-0cb9-45c9-970e-6607ba087a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:36.442638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.442609 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d636b736-wjrmg\" (UID: \"2ed55dba-0cb9-45c9-970e-6607ba087a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:36.464162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.464126 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfhjc\" (UniqueName: \"kubernetes.io/projected/2ed55dba-0cb9-45c9-970e-6607ba087a21-kube-api-access-xfhjc\") pod \"maas-api-revoke-sub-d636b736-wjrmg\" (UID: \"2ed55dba-0cb9-45c9-970e-6607ba087a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:36.750676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:36.750592 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:37.061801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.061769 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg"] Oct 30 18:28:37.062392 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:28:37.062360 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed55dba_0cb9_45c9_970e_6607ba087a21.slice/crio-a4fee68b94051364e8fc052d6a17bb71d81d89344a97c68789fc60ab4545c2c4 WatchSource:0}: Error finding container a4fee68b94051364e8fc052d6a17bb71d81d89344a97c68789fc60ab4545c2c4: Status 404 returned error can't find the container with id a4fee68b94051364e8fc052d6a17bb71d81d89344a97c68789fc60ab4545c2c4 Oct 30 18:28:37.410215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.410171 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" event={"ID":"2ed55dba-0cb9-45c9-970e-6607ba087a21","Type":"ContainerStarted","Data":"f08c4ba2fa8908df9aaf58257c378d5e2190e4fdf6d4dfa8b2572d3241b64b93"} Oct 30 18:28:37.410682 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.410230 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" event={"ID":"2ed55dba-0cb9-45c9-970e-6607ba087a21","Type":"ContainerStarted","Data":"a4fee68b94051364e8fc052d6a17bb71d81d89344a97c68789fc60ab4545c2c4"} Oct 30 18:28:37.456287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.456235 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" podStartSLOduration=1.4562199599999999 podStartE2EDuration="1.45621996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:37.453433809 +0000 UTC m=+2176.410437647" watchObservedRunningTime="2026-10-30 18:28:37.45621996 +0000 UTC m=+2176.413223844" Oct 30 18:28:37.825440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.825363 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg"] Oct 30 18:28:37.829720 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.829701 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:37.850348 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.850310 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg"] Oct 30 18:28:37.857690 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.857640 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b8b7dd-ddwjg\" (UID: \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:37.857854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.857776 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b8b7dd-ddwjg\" (UID: \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:37.858033 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.858008 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpx86\" (UniqueName: \"kubernetes.io/projected/c2555b8c-3a91-4de0-892f-af62a83a2b5c-kube-api-access-rpx86\") pod \"maas-api-revoke-sub-a8b8b7dd-ddwjg\" (UID: \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:37.959767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.959733 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b8b7dd-ddwjg\" (UID: \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:37.959974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.959791 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b8b7dd-ddwjg\" (UID: \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:37.959974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.959902 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpx86\" (UniqueName: \"kubernetes.io/projected/c2555b8c-3a91-4de0-892f-af62a83a2b5c-kube-api-access-rpx86\") pod \"maas-api-revoke-sub-a8b8b7dd-ddwjg\" (UID: \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:37.960453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.960430 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b8b7dd-ddwjg\" (UID: \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:37.962298 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.962277 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b8b7dd-ddwjg\" (UID: \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:37.984886 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:37.984857 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpx86\" (UniqueName: \"kubernetes.io/projected/c2555b8c-3a91-4de0-892f-af62a83a2b5c-kube-api-access-rpx86\") pod \"maas-api-revoke-sub-a8b8b7dd-ddwjg\" (UID: \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:38.038858 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:28:38.038828 4883 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed55dba_0cb9_45c9_970e_6607ba087a21.slice/crio-f08c4ba2fa8908df9aaf58257c378d5e2190e4fdf6d4dfa8b2572d3241b64b93.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:28:38.144905 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:38.144870 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:38.428561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:38.428470 4883 generic.go:356] "Generic (PLEG): container finished" podID="2ed55dba-0cb9-45c9-970e-6607ba087a21" containerID="f08c4ba2fa8908df9aaf58257c378d5e2190e4fdf6d4dfa8b2572d3241b64b93" exitCode=0 Oct 30 18:28:38.429057 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:38.428554 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" event={"ID":"2ed55dba-0cb9-45c9-970e-6607ba087a21","Type":"ContainerDied","Data":"f08c4ba2fa8908df9aaf58257c378d5e2190e4fdf6d4dfa8b2572d3241b64b93"} Oct 30 18:28:38.475780 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:38.475744 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg"] Oct 30 18:28:38.476576 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:28:38.476536 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2555b8c_3a91_4de0_892f_af62a83a2b5c.slice/crio-254b6a622b1da4747846da5f7c7ac730f32937dbc89dbff5cbcb755097ff0126 WatchSource:0}: Error finding container 254b6a622b1da4747846da5f7c7ac730f32937dbc89dbff5cbcb755097ff0126: Status 404 returned error can't find the container with id 254b6a622b1da4747846da5f7c7ac730f32937dbc89dbff5cbcb755097ff0126 Oct 30 18:28:39.438103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.438067 4883 generic.go:356] "Generic (PLEG): container finished" podID="c2555b8c-3a91-4de0-892f-af62a83a2b5c" containerID="0a43171a0de352061d451fdb98be646803aca5fd365f3504eda3b1442437861c" exitCode=0 Oct 30 18:28:39.438586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.438154 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" event={"ID":"c2555b8c-3a91-4de0-892f-af62a83a2b5c","Type":"ContainerDied","Data":"0a43171a0de352061d451fdb98be646803aca5fd365f3504eda3b1442437861c"} Oct 30 18:28:39.438586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.438198 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" event={"ID":"c2555b8c-3a91-4de0-892f-af62a83a2b5c","Type":"ContainerStarted","Data":"254b6a622b1da4747846da5f7c7ac730f32937dbc89dbff5cbcb755097ff0126"} Oct 30 18:28:39.608545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.608520 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:39.687126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.687038 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-openshift-ca\") pod \"2ed55dba-0cb9-45c9-970e-6607ba087a21\" (UID: \"2ed55dba-0cb9-45c9-970e-6607ba087a21\") " Oct 30 18:28:39.687281 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.687200 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed55dba-0cb9-45c9-970e-6607ba087a21-kube-api-access-xfhjc\" (UniqueName: \"kubernetes.io/projected/2ed55dba-0cb9-45c9-970e-6607ba087a21-kube-api-access-xfhjc\") pod \"2ed55dba-0cb9-45c9-970e-6607ba087a21\" (UID: \"2ed55dba-0cb9-45c9-970e-6607ba087a21\") " Oct 30 18:28:39.687359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.687290 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-xks-ca\") pod \"2ed55dba-0cb9-45c9-970e-6607ba087a21\" (UID: \"2ed55dba-0cb9-45c9-970e-6607ba087a21\") " Oct 30 18:28:39.687731 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.687707 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-openshift-ca" pod "2ed55dba-0cb9-45c9-970e-6607ba087a21" (UID: "2ed55dba-0cb9-45c9-970e-6607ba087a21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:39.687963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.687916 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:39.689571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.689548 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed55dba-0cb9-45c9-970e-6607ba087a21-kube-api-access-xfhjc" pod "2ed55dba-0cb9-45c9-970e-6607ba087a21" (UID: "2ed55dba-0cb9-45c9-970e-6607ba087a21"). InnerVolumeSpecName "kube-api-access-xfhjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:39.689653 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.689565 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-xks-ca" pod "2ed55dba-0cb9-45c9-970e-6607ba087a21" (UID: "2ed55dba-0cb9-45c9-970e-6607ba087a21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:39.789487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.789447 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed55dba-0cb9-45c9-970e-6607ba087a21-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:39.789487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:39.789480 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfhjc\" (UniqueName: \"kubernetes.io/projected/2ed55dba-0cb9-45c9-970e-6607ba087a21-kube-api-access-xfhjc\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:40.454173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.454098 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" Oct 30 18:28:40.454669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.454493 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg" event={"ID":"2ed55dba-0cb9-45c9-970e-6607ba087a21","Type":"ContainerDied","Data":"a4fee68b94051364e8fc052d6a17bb71d81d89344a97c68789fc60ab4545c2c4"} Oct 30 18:28:40.454669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.454544 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4fee68b94051364e8fc052d6a17bb71d81d89344a97c68789fc60ab4545c2c4" Oct 30 18:28:40.627894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.627866 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:40.701355 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.701317 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2555b8c-3a91-4de0-892f-af62a83a2b5c-kube-api-access-rpx86\" (UniqueName: \"kubernetes.io/projected/c2555b8c-3a91-4de0-892f-af62a83a2b5c-kube-api-access-rpx86\") pod \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\" (UID: \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\") " Oct 30 18:28:40.701557 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.701374 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-xks-ca\") pod \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\" (UID: \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\") " Oct 30 18:28:40.701670 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.701649 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-openshift-ca\") pod \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\" (UID: \"c2555b8c-3a91-4de0-892f-af62a83a2b5c\") " Oct 30 18:28:40.702066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.702032 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-openshift-ca" pod "c2555b8c-3a91-4de0-892f-af62a83a2b5c" (UID: "c2555b8c-3a91-4de0-892f-af62a83a2b5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:40.702282 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.702261 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:40.703817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.703793 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-xks-ca" pod "c2555b8c-3a91-4de0-892f-af62a83a2b5c" (UID: "c2555b8c-3a91-4de0-892f-af62a83a2b5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:40.704229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.704173 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2555b8c-3a91-4de0-892f-af62a83a2b5c-kube-api-access-rpx86" pod "c2555b8c-3a91-4de0-892f-af62a83a2b5c" (UID: "c2555b8c-3a91-4de0-892f-af62a83a2b5c"). InnerVolumeSpecName "kube-api-access-rpx86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:40.803806 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.803767 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpx86\" (UniqueName: \"kubernetes.io/projected/c2555b8c-3a91-4de0-892f-af62a83a2b5c-kube-api-access-rpx86\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:40.803806 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:40.803801 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2555b8c-3a91-4de0-892f-af62a83a2b5c-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:41.462436 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:41.462344 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" Oct 30 18:28:41.462436 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:41.462366 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg" event={"ID":"c2555b8c-3a91-4de0-892f-af62a83a2b5c","Type":"ContainerDied","Data":"254b6a622b1da4747846da5f7c7ac730f32937dbc89dbff5cbcb755097ff0126"} Oct 30 18:28:41.462864 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:41.462455 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="254b6a622b1da4747846da5f7c7ac730f32937dbc89dbff5cbcb755097ff0126" Oct 30 18:28:53.729626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.729586 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w"] Oct 30 18:28:53.730180 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.730166 4883 cpu_manager.go:414] "containerMap: removing container" podUID="c2555b8c-3a91-4de0-892f-af62a83a2b5c" containerName="revoke-keys" Oct 30 18:28:53.730225 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.730181 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2555b8c-3a91-4de0-892f-af62a83a2b5c" containerName="revoke-keys" Oct 30 18:28:53.730225 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.730193 4883 cpu_manager.go:414] "containerMap: removing container" podUID="2ed55dba-0cb9-45c9-970e-6607ba087a21" containerName="revoke-keys" Oct 30 18:28:53.730225 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.730198 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed55dba-0cb9-45c9-970e-6607ba087a21" containerName="revoke-keys" Oct 30 18:28:53.730321 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.730298 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2555b8c-3a91-4de0-892f-af62a83a2b5c" containerName="revoke-keys" Oct 30 18:28:53.730321 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.730306 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed55dba-0cb9-45c9-970e-6607ba087a21" containerName="revoke-keys" Oct 30 18:28:53.733586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.733563 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:28:53.766745 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.766701 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w"] Oct 30 18:28:53.780291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.777629 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:28:53.780291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.777764 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:28:53.780291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.777801 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:53.780291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.777856 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:28:53.897528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.897492 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04839d99-vmn5w\" (UID: \"5201f205-e45c-4fc6-955d-8a65329dcfca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:28:53.897703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.897675 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04839d99-vmn5w\" (UID: \"5201f205-e45c-4fc6-955d-8a65329dcfca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:28:53.897744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.897722 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm7vx\" (UniqueName: \"kubernetes.io/projected/5201f205-e45c-4fc6-955d-8a65329dcfca-kube-api-access-lm7vx\") pod \"maas-api-revoke-sub-04839d99-vmn5w\" (UID: \"5201f205-e45c-4fc6-955d-8a65329dcfca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:28:53.999803 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.999709 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04839d99-vmn5w\" (UID: \"5201f205-e45c-4fc6-955d-8a65329dcfca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:28:54.000005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.999852 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04839d99-vmn5w\" (UID: \"5201f205-e45c-4fc6-955d-8a65329dcfca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:28:54.000005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:53.999886 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm7vx\" (UniqueName: \"kubernetes.io/projected/5201f205-e45c-4fc6-955d-8a65329dcfca-kube-api-access-lm7vx\") pod \"maas-api-revoke-sub-04839d99-vmn5w\" (UID: \"5201f205-e45c-4fc6-955d-8a65329dcfca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:28:54.000568 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:54.000544 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04839d99-vmn5w\" (UID: \"5201f205-e45c-4fc6-955d-8a65329dcfca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:28:54.002138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:54.002115 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04839d99-vmn5w\" (UID: \"5201f205-e45c-4fc6-955d-8a65329dcfca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:28:54.024798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:54.024771 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm7vx\" (UniqueName: \"kubernetes.io/projected/5201f205-e45c-4fc6-955d-8a65329dcfca-kube-api-access-lm7vx\") pod \"maas-api-revoke-sub-04839d99-vmn5w\" (UID: \"5201f205-e45c-4fc6-955d-8a65329dcfca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:28:54.048771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:54.048743 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:28:54.355756 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:54.355726 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w"] Oct 30 18:28:54.356266 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:28:54.356233 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5201f205_e45c_4fc6_955d_8a65329dcfca.slice/crio-7d031c85893cb3d70cc3b10dfd866f0ac2e3a40247b6cba042c3e56dca2c98f4 WatchSource:0}: Error finding container 7d031c85893cb3d70cc3b10dfd866f0ac2e3a40247b6cba042c3e56dca2c98f4: Status 404 returned error can't find the container with id 7d031c85893cb3d70cc3b10dfd866f0ac2e3a40247b6cba042c3e56dca2c98f4 Oct 30 18:28:54.579449 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:54.579416 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" event={"ID":"5201f205-e45c-4fc6-955d-8a65329dcfca","Type":"ContainerStarted","Data":"7d031c85893cb3d70cc3b10dfd866f0ac2e3a40247b6cba042c3e56dca2c98f4"} Oct 30 18:28:55.591279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:55.591189 4883 generic.go:356] "Generic (PLEG): container finished" podID="5201f205-e45c-4fc6-955d-8a65329dcfca" containerID="e0a90f5772055ff3c8ec3c229fbb2ce2a1115ee55495c9f28327df219644870a" exitCode=0 Oct 30 18:28:55.591279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:55.591231 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" event={"ID":"5201f205-e45c-4fc6-955d-8a65329dcfca","Type":"ContainerDied","Data":"e0a90f5772055ff3c8ec3c229fbb2ce2a1115ee55495c9f28327df219644870a"} Oct 30 18:28:56.771213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:56.771180 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:28:56.839785 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:56.839746 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5201f205-e45c-4fc6-955d-8a65329dcfca-kube-api-access-lm7vx\" (UniqueName: \"kubernetes.io/projected/5201f205-e45c-4fc6-955d-8a65329dcfca-kube-api-access-lm7vx\") pod \"5201f205-e45c-4fc6-955d-8a65329dcfca\" (UID: \"5201f205-e45c-4fc6-955d-8a65329dcfca\") " Oct 30 18:28:56.840009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:56.839842 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-openshift-ca\") pod \"5201f205-e45c-4fc6-955d-8a65329dcfca\" (UID: \"5201f205-e45c-4fc6-955d-8a65329dcfca\") " Oct 30 18:28:56.840009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:56.839905 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-xks-ca\") pod \"5201f205-e45c-4fc6-955d-8a65329dcfca\" (UID: \"5201f205-e45c-4fc6-955d-8a65329dcfca\") " Oct 30 18:28:56.840258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:56.840233 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-openshift-ca" pod "5201f205-e45c-4fc6-955d-8a65329dcfca" (UID: "5201f205-e45c-4fc6-955d-8a65329dcfca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:56.840426 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:56.840410 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:56.842368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:56.842341 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5201f205-e45c-4fc6-955d-8a65329dcfca-kube-api-access-lm7vx" pod "5201f205-e45c-4fc6-955d-8a65329dcfca" (UID: "5201f205-e45c-4fc6-955d-8a65329dcfca"). InnerVolumeSpecName "kube-api-access-lm7vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:56.844146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:56.844124 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-xks-ca" pod "5201f205-e45c-4fc6-955d-8a65329dcfca" (UID: "5201f205-e45c-4fc6-955d-8a65329dcfca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:56.941895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:56.941853 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm7vx\" (UniqueName: \"kubernetes.io/projected/5201f205-e45c-4fc6-955d-8a65329dcfca-kube-api-access-lm7vx\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:56.941895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:56.941892 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5201f205-e45c-4fc6-955d-8a65329dcfca-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:28:57.609335 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:57.609293 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" event={"ID":"5201f205-e45c-4fc6-955d-8a65329dcfca","Type":"ContainerDied","Data":"7d031c85893cb3d70cc3b10dfd866f0ac2e3a40247b6cba042c3e56dca2c98f4"} Oct 30 18:28:57.609527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:57.609363 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d031c85893cb3d70cc3b10dfd866f0ac2e3a40247b6cba042c3e56dca2c98f4" Oct 30 18:28:57.609527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:28:57.609308 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w" Oct 30 18:29:05.097255 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:29:05.097162 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:29:06.576845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.576809 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb"] Oct 30 18:29:06.577407 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.577391 4883 cpu_manager.go:414] "containerMap: removing container" podUID="5201f205-e45c-4fc6-955d-8a65329dcfca" containerName="revoke-keys" Oct 30 18:29:06.577452 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.577408 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5201f205-e45c-4fc6-955d-8a65329dcfca" containerName="revoke-keys" Oct 30 18:29:06.577521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.577511 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="5201f205-e45c-4fc6-955d-8a65329dcfca" containerName="revoke-keys" Oct 30 18:29:06.581162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.581143 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:06.597123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.597092 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16bcb652-97tnb\" (UID: \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:06.597287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.597266 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16bcb652-97tnb\" (UID: \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:06.597562 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.597539 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxg2l\" (UniqueName: \"kubernetes.io/projected/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-kube-api-access-zxg2l\") pod \"maas-api-revoke-sub-16bcb652-97tnb\" (UID: \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:06.602495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.602468 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb"] Oct 30 18:29:06.610551 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.610526 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:29:06.610705 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.610573 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:29:06.610705 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.610621 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:29:06.610705 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.610681 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:29:06.698613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.698580 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16bcb652-97tnb\" (UID: \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:06.698787 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.698713 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxg2l\" (UniqueName: \"kubernetes.io/projected/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-kube-api-access-zxg2l\") pod \"maas-api-revoke-sub-16bcb652-97tnb\" (UID: \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:06.698787 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.698778 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16bcb652-97tnb\" (UID: \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:06.699292 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.699269 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16bcb652-97tnb\" (UID: \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:06.701373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.701347 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16bcb652-97tnb\" (UID: \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:06.724271 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.724245 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxg2l\" (UniqueName: \"kubernetes.io/projected/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-kube-api-access-zxg2l\") pod \"maas-api-revoke-sub-16bcb652-97tnb\" (UID: \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:06.902475 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:06.902175 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:07.248696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:07.248667 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb"] Oct 30 18:29:07.249060 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:29:07.249036 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fea8dd3_63ff_4471_8e0c_53bc6b0b803f.slice/crio-2239bef8971b437699f0993f12fe66f69e70156a2327c8bcd3be7c7141f38d29 WatchSource:0}: Error finding container 2239bef8971b437699f0993f12fe66f69e70156a2327c8bcd3be7c7141f38d29: Status 404 returned error can't find the container with id 2239bef8971b437699f0993f12fe66f69e70156a2327c8bcd3be7c7141f38d29 Oct 30 18:29:07.694342 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:07.694300 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" event={"ID":"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f","Type":"ContainerStarted","Data":"470d1dd7d61fcbff7068d0cdd69cb2f374a82b0b638ccdaa2096b00b015c47ba"} Oct 30 18:29:07.694840 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:07.694811 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" event={"ID":"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f","Type":"ContainerStarted","Data":"2239bef8971b437699f0993f12fe66f69e70156a2327c8bcd3be7c7141f38d29"} Oct 30 18:29:07.747462 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:07.747406 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" podStartSLOduration=1.7473876640000001 podStartE2EDuration="1.747387664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:07.746092708 +0000 UTC m=+2206.703096545" watchObservedRunningTime="2026-10-30 18:29:07.747387664 +0000 UTC m=+2206.704391501" Oct 30 18:29:08.703749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:08.703712 4883 generic.go:356] "Generic (PLEG): container finished" podID="4fea8dd3-63ff-4471-8e0c-53bc6b0b803f" containerID="470d1dd7d61fcbff7068d0cdd69cb2f374a82b0b638ccdaa2096b00b015c47ba" exitCode=0 Oct 30 18:29:08.704194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:08.703793 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" event={"ID":"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f","Type":"ContainerDied","Data":"470d1dd7d61fcbff7068d0cdd69cb2f374a82b0b638ccdaa2096b00b015c47ba"} Oct 30 18:29:09.878789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:09.878764 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:09.963312 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:09.963274 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-xks-ca\") pod \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\" (UID: \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\") " Oct 30 18:29:09.963521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:09.963370 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-openshift-ca\") pod \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\" (UID: \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\") " Oct 30 18:29:09.963521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:09.963442 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-kube-api-access-zxg2l\" (UniqueName: \"kubernetes.io/projected/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-kube-api-access-zxg2l\") pod \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\" (UID: \"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f\") " Oct 30 18:29:09.963814 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:09.963783 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-openshift-ca" pod "4fea8dd3-63ff-4471-8e0c-53bc6b0b803f" (UID: "4fea8dd3-63ff-4471-8e0c-53bc6b0b803f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:09.964145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:09.964120 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:29:09.966221 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:09.966146 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-xks-ca" pod "4fea8dd3-63ff-4471-8e0c-53bc6b0b803f" (UID: "4fea8dd3-63ff-4471-8e0c-53bc6b0b803f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:09.966330 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:09.966262 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-kube-api-access-zxg2l" pod "4fea8dd3-63ff-4471-8e0c-53bc6b0b803f" (UID: "4fea8dd3-63ff-4471-8e0c-53bc6b0b803f"). InnerVolumeSpecName "kube-api-access-zxg2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:10.065727 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:10.065692 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:29:10.065727 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:10.065723 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxg2l\" (UniqueName: \"kubernetes.io/projected/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f-kube-api-access-zxg2l\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:29:10.725139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:10.725104 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" Oct 30 18:29:10.725322 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:10.725136 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb" event={"ID":"4fea8dd3-63ff-4471-8e0c-53bc6b0b803f","Type":"ContainerDied","Data":"2239bef8971b437699f0993f12fe66f69e70156a2327c8bcd3be7c7141f38d29"} Oct 30 18:29:10.725322 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:10.725188 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2239bef8971b437699f0993f12fe66f69e70156a2327c8bcd3be7c7141f38d29" Oct 30 18:29:29.508320 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.508279 4883 scope.go:122] "RemoveContainer" containerID="626f18d66ed1dd8775a84631877c105b003142ae3065bc30d842bbe3a3fc52a2" Oct 30 18:29:29.532575 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.532543 4883 scope.go:122] "RemoveContainer" containerID="6f050e576fefdd5b6274e2e176a982096019c6c1bdd7d6c2ab5def663387bba8" Oct 30 18:29:29.761275 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.761189 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9"] Oct 30 18:29:29.762022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.761993 4883 cpu_manager.go:414] "containerMap: removing container" podUID="4fea8dd3-63ff-4471-8e0c-53bc6b0b803f" containerName="revoke-keys" Oct 30 18:29:29.763454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.763004 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fea8dd3-63ff-4471-8e0c-53bc6b0b803f" containerName="revoke-keys" Oct 30 18:29:29.763454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.763300 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fea8dd3-63ff-4471-8e0c-53bc6b0b803f" containerName="revoke-keys" Oct 30 18:29:29.769109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.769085 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:29.788155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.788130 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-5cb3d5-openshift-default-dockercfg-69d2c\"" Oct 30 18:29:29.806996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.806965 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9"] Oct 30 18:29:29.913651 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.913619 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-envoy\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:29.914036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.913994 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istiod-ca-cert\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:29.914309 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.914290 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-socket\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:29.914590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.914569 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-certs\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:29.914753 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.914738 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-data\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:29.914973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.914956 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-podinfo\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:29.915129 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.915111 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjq9\" (UniqueName: \"kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-kube-api-access-hwjq9\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:29.915405 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.915384 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-credential-socket\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:29.915615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:29.915599 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-token\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.018866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.018765 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-data\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.019200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.019167 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-podinfo\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.019334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.019211 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-data\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.019334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.019220 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjq9\" (UniqueName: \"kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-kube-api-access-hwjq9\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.019468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.019349 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-credential-socket\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.019468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.019438 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-token\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.019580 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.019563 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-envoy\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.019638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.019589 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istiod-ca-cert\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.019694 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.019656 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-socket\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.019754 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.019742 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-certs\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.020261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.020239 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-certs\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.020426 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.020398 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-socket\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.020512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.020299 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-credential-socket\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.020677 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.020655 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istiod-ca-cert\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.023950 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.023910 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-podinfo\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.025168 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.025146 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-envoy\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.046280 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.046239 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-token\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.046878 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.046727 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjq9\" (UniqueName: \"kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-kube-api-access-hwjq9\") pod \"e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.101696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.101661 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:30.437042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.437007 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9"] Oct 30 18:29:30.437620 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:29:30.437593 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bb0b62_59e5_4aa2_b5d1_c8837731bf53.slice/crio-eed7a1dc338254a0cb0fae4b05819f2cf1babaad257f3f6ea51743b7517e90ac WatchSource:0}: Error finding container eed7a1dc338254a0cb0fae4b05819f2cf1babaad257f3f6ea51743b7517e90ac: Status 404 returned error can't find the container with id eed7a1dc338254a0cb0fae4b05819f2cf1babaad257f3f6ea51743b7517e90ac Oct 30 18:29:30.440633 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.440589 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:29:30.440731 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.440681 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:29:30.440731 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.440722 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:29:30.909305 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.909267 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" event={"ID":"63bb0b62-59e5-4aa2-b5d1-c8837731bf53","Type":"ContainerStarted","Data":"40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181"} Oct 30 18:29:30.909720 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.909353 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" event={"ID":"63bb0b62-59e5-4aa2-b5d1-c8837731bf53","Type":"ContainerStarted","Data":"eed7a1dc338254a0cb0fae4b05819f2cf1babaad257f3f6ea51743b7517e90ac"} Oct 30 18:29:30.973473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:30.973419 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" podStartSLOduration=1.973403903 podStartE2EDuration="1.973403903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:30.969778129 +0000 UTC m=+2229.926781966" watchObservedRunningTime="2026-10-30 18:29:30.973403903 +0000 UTC m=+2229.930407738" Oct 30 18:29:31.102128 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:31.102088 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:31.109727 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:31.109699 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:31.916223 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:31.916186 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:31.917268 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:29:31.917245 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:29:35.087517 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:29:35.087480 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:30:00.160467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.160424 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx"] Oct 30 18:30:00.168081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.168048 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" Oct 30 18:30:00.169066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.169037 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj"] Oct 30 18:30:00.175969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.175441 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" Oct 30 18:30:00.175969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.175570 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8"] Oct 30 18:30:00.198142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.198113 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:30:00.199208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.199176 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:30:00.199485 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.199456 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:30:00.210349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.210321 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz"] Oct 30 18:30:00.212223 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.211714 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" Oct 30 18:30:00.276956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.262444 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj"] Oct 30 18:30:00.276956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.262496 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx"] Oct 30 18:30:00.276956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.262598 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" Oct 30 18:30:00.319965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.309860 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpfbz\" (UniqueName: \"kubernetes.io/projected/96a19637-c44e-414a-b3b2-d963cf2ec937-kube-api-access-tpfbz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8\" (UID: \"96a19637-c44e-414a-b3b2-d963cf2ec937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" Oct 30 18:30:00.319965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.310007 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj\" (UID: \"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" Oct 30 18:30:00.327395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.320294 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1041e24a-32c1-49ff-9c78-a1708404d7fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx\" (UID: \"1041e24a-32c1-49ff-9c78-a1708404d7fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" Oct 30 18:30:00.327395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.320392 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x294j\" (UniqueName: \"kubernetes.io/projected/1041e24a-32c1-49ff-9c78-a1708404d7fd-kube-api-access-x294j\") pod \"maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx\" (UID: \"1041e24a-32c1-49ff-9c78-a1708404d7fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" Oct 30 18:30:00.327395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.320544 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a19637-c44e-414a-b3b2-d963cf2ec937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8\" (UID: \"96a19637-c44e-414a-b3b2-d963cf2ec937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" Oct 30 18:30:00.327395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.320688 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69lqp\" (UniqueName: \"kubernetes.io/projected/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-kube-api-access-69lqp\") pod \"maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj\" (UID: \"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" Oct 30 18:30:00.342552 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.342503 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8"] Oct 30 18:30:00.352423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.352394 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj"] Oct 30 18:30:00.411295 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.402504 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" Oct 30 18:30:00.441036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.421695 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0213bbf6-2490-46de-a522-f12d2f4f2515-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz\" (UID: \"0213bbf6-2490-46de-a522-f12d2f4f2515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" Oct 30 18:30:00.441036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.421881 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69lqp\" (UniqueName: \"kubernetes.io/projected/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-kube-api-access-69lqp\") pod \"maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj\" (UID: \"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" Oct 30 18:30:00.441036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.421970 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpfbz\" (UniqueName: \"kubernetes.io/projected/96a19637-c44e-414a-b3b2-d963cf2ec937-kube-api-access-tpfbz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8\" (UID: \"96a19637-c44e-414a-b3b2-d963cf2ec937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" Oct 30 18:30:00.441036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.422008 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fgvq\" (UniqueName: \"kubernetes.io/projected/0213bbf6-2490-46de-a522-f12d2f4f2515-kube-api-access-6fgvq\") pod \"maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz\" (UID: \"0213bbf6-2490-46de-a522-f12d2f4f2515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" Oct 30 18:30:00.441036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.422071 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj\" (UID: \"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" Oct 30 18:30:00.441036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.422102 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1041e24a-32c1-49ff-9c78-a1708404d7fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx\" (UID: \"1041e24a-32c1-49ff-9c78-a1708404d7fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" Oct 30 18:30:00.441036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.422155 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x294j\" (UniqueName: \"kubernetes.io/projected/1041e24a-32c1-49ff-9c78-a1708404d7fd-kube-api-access-x294j\") pod \"maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx\" (UID: \"1041e24a-32c1-49ff-9c78-a1708404d7fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" Oct 30 18:30:00.441036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.422273 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a19637-c44e-414a-b3b2-d963cf2ec937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8\" (UID: \"96a19637-c44e-414a-b3b2-d963cf2ec937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" Oct 30 18:30:00.441036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.423248 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1041e24a-32c1-49ff-9c78-a1708404d7fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx\" (UID: \"1041e24a-32c1-49ff-9c78-a1708404d7fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" Oct 30 18:30:00.441036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.423882 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj\" (UID: \"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" Oct 30 18:30:00.441036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.425363 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a19637-c44e-414a-b3b2-d963cf2ec937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8\" (UID: \"96a19637-c44e-414a-b3b2-d963cf2ec937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" Oct 30 18:30:00.453258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.453227 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h"] Oct 30 18:30:00.466457 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.466427 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz"] Oct 30 18:30:00.467021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.466885 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" Oct 30 18:30:00.561958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.532080 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgvq\" (UniqueName: \"kubernetes.io/projected/0213bbf6-2490-46de-a522-f12d2f4f2515-kube-api-access-6fgvq\") pod \"maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz\" (UID: \"0213bbf6-2490-46de-a522-f12d2f4f2515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" Oct 30 18:30:00.561958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.532328 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0213bbf6-2490-46de-a522-f12d2f4f2515-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz\" (UID: \"0213bbf6-2490-46de-a522-f12d2f4f2515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" Oct 30 18:30:00.561958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.533279 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0213bbf6-2490-46de-a522-f12d2f4f2515-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz\" (UID: \"0213bbf6-2490-46de-a522-f12d2f4f2515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" Oct 30 18:30:00.561958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.537608 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69lqp\" (UniqueName: \"kubernetes.io/projected/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-kube-api-access-69lqp\") pod \"maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj\" (UID: \"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" Oct 30 18:30:00.561958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.537897 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj"] Oct 30 18:30:00.720009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.719741 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" Oct 30 18:30:00.785972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.777423 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888e9299-97b3-4cef-9213-69663ac795b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj\" (UID: \"888e9299-97b3-4cef-9213-69663ac795b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" Oct 30 18:30:00.785972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.777634 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg7c8\" (UniqueName: \"kubernetes.io/projected/77be19c9-c897-46ac-bcfc-da3e3623e2f5-kube-api-access-fg7c8\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h\" (UID: \"77be19c9-c897-46ac-bcfc-da3e3623e2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" Oct 30 18:30:00.785972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.777667 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctv8s\" (UniqueName: \"kubernetes.io/projected/888e9299-97b3-4cef-9213-69663ac795b0-kube-api-access-ctv8s\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj\" (UID: \"888e9299-97b3-4cef-9213-69663ac795b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" Oct 30 18:30:00.785972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.777796 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77be19c9-c897-46ac-bcfc-da3e3623e2f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h\" (UID: \"77be19c9-c897-46ac-bcfc-da3e3623e2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" Oct 30 18:30:00.911550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.881037 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg7c8\" (UniqueName: \"kubernetes.io/projected/77be19c9-c897-46ac-bcfc-da3e3623e2f5-kube-api-access-fg7c8\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h\" (UID: \"77be19c9-c897-46ac-bcfc-da3e3623e2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" Oct 30 18:30:00.911550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.881090 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctv8s\" (UniqueName: \"kubernetes.io/projected/888e9299-97b3-4cef-9213-69663ac795b0-kube-api-access-ctv8s\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj\" (UID: \"888e9299-97b3-4cef-9213-69663ac795b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" Oct 30 18:30:00.911550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.881230 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77be19c9-c897-46ac-bcfc-da3e3623e2f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h\" (UID: \"77be19c9-c897-46ac-bcfc-da3e3623e2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" Oct 30 18:30:00.911550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.881326 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888e9299-97b3-4cef-9213-69663ac795b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj\" (UID: \"888e9299-97b3-4cef-9213-69663ac795b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" Oct 30 18:30:00.911550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.882259 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888e9299-97b3-4cef-9213-69663ac795b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj\" (UID: \"888e9299-97b3-4cef-9213-69663ac795b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" Oct 30 18:30:00.911550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.883574 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77be19c9-c897-46ac-bcfc-da3e3623e2f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h\" (UID: \"77be19c9-c897-46ac-bcfc-da3e3623e2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" Oct 30 18:30:00.911550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.888426 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" Oct 30 18:30:00.911550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.894158 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x294j\" (UniqueName: \"kubernetes.io/projected/1041e24a-32c1-49ff-9c78-a1708404d7fd-kube-api-access-x294j\") pod \"maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx\" (UID: \"1041e24a-32c1-49ff-9c78-a1708404d7fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" Oct 30 18:30:00.916843 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.916814 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj"] Oct 30 18:30:00.990963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.989522 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj\" (UID: \"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" Oct 30 18:30:00.990963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:00.989653 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzbm\" (UniqueName: \"kubernetes.io/projected/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-kube-api-access-7zzbm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj\" (UID: \"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" Oct 30 18:30:01.088025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.060990 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h"] Oct 30 18:30:01.088025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.075150 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgvq\" (UniqueName: \"kubernetes.io/projected/0213bbf6-2490-46de-a522-f12d2f4f2515-kube-api-access-6fgvq\") pod \"maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz\" (UID: \"0213bbf6-2490-46de-a522-f12d2f4f2515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" Oct 30 18:30:01.088025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.076638 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpfbz\" (UniqueName: \"kubernetes.io/projected/96a19637-c44e-414a-b3b2-d963cf2ec937-kube-api-access-tpfbz\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8\" (UID: \"96a19637-c44e-414a-b3b2-d963cf2ec937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" Oct 30 18:30:01.112209 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.091839 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzbm\" (UniqueName: \"kubernetes.io/projected/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-kube-api-access-7zzbm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj\" (UID: \"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" Oct 30 18:30:01.113497 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.113237 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg7c8\" (UniqueName: \"kubernetes.io/projected/77be19c9-c897-46ac-bcfc-da3e3623e2f5-kube-api-access-fg7c8\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h\" (UID: \"77be19c9-c897-46ac-bcfc-da3e3623e2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" Oct 30 18:30:01.124140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.124107 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctv8s\" (UniqueName: \"kubernetes.io/projected/888e9299-97b3-4cef-9213-69663ac795b0-kube-api-access-ctv8s\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj\" (UID: \"888e9299-97b3-4cef-9213-69663ac795b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" Oct 30 18:30:01.256292 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.247498 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" Oct 30 18:30:01.295531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.294217 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" Oct 30 18:30:01.295531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.265840 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj\" (UID: \"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" Oct 30 18:30:01.295531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.266518 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj\" (UID: \"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" Oct 30 18:30:01.295531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.295445 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" Oct 30 18:30:01.295531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.265477 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" Oct 30 18:30:01.337983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.337765 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzbm\" (UniqueName: \"kubernetes.io/projected/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-kube-api-access-7zzbm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj\" (UID: \"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" Oct 30 18:30:01.358841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.358803 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" Oct 30 18:30:01.572274 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.571104 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:30:01.572274 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.571251 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:01.615960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.615892 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:01.618459 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.618429 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" Oct 30 18:30:01.794893 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.794790 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:30:01.810169 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:01.810123 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:02.059015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:02.049839 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7"] Oct 30 18:30:02.078110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:02.078063 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:02.102581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:02.102548 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" Oct 30 18:30:02.209431 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:02.209397 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj"] Oct 30 18:30:02.267702 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:02.267656 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp2mz\" (UniqueName: \"kubernetes.io/projected/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-kube-api-access-cp2mz\") pod \"maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7\" (UID: \"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" Oct 30 18:30:02.273258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:02.273226 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7\" (UID: \"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" Oct 30 18:30:02.396757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:02.386956 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp2mz\" (UniqueName: \"kubernetes.io/projected/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-kube-api-access-cp2mz\") pod \"maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7\" (UID: \"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" Oct 30 18:30:02.396757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:02.387083 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7\" (UID: \"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" Oct 30 18:30:02.514186 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:02.514144 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7\" (UID: \"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" Oct 30 18:30:02.658237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:02.658049 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7"] Oct 30 18:30:02.706510 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:02.706466 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp2mz\" (UniqueName: \"kubernetes.io/projected/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-kube-api-access-cp2mz\") pod \"maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7\" (UID: \"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" Oct 30 18:30:02.874633 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:02.874344 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" Oct 30 18:30:03.145006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.107578 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:03.145006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.134755 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:03.145006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.137969 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:30:03.145006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.138270 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:30:03.145006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.138509 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:30:03.145006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.138751 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:03.177995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.147661 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:30:03.177995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.148143 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:30:03.177995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.149596 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:30:03.196496 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.196033 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:03.196496 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.196099 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:03.208241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.208132 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:03.319376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.319115 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:03.355844 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.320013 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:30:04.004007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:03.987729 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.070182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.070136 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" event={"ID":"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607","Type":"ContainerStarted","Data":"d5aa5f5de8d920b558028f3949b6a84ddd65e6a5ed83d2a3e81b310b42c1cd2b"} Oct 30 18:30:04.127306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.127264 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:30:04.127695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.127676 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:30:04.127885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.127864 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:30:04.135958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.128029 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:30:04.251958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.154055 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:30:04.251958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.154633 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:30:04.252649 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.252039 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.252649 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.252359 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:30:04.252649 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.252597 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.252838 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.252787 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.272210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.272099 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.279983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.276091 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:04.316994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.316953 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.350125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.350070 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.411458 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.354024 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:30:04.462261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.448150 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:30:04.462261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.448241 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:30:04.462261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.448300 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:30:04.596866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.590623 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:30:04.596866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.590853 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:30:04.679565 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.674097 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:04.992574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:04.992239 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:30:05.056958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:05.033736 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.072960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:05.061536 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:30:05.156970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:05.155031 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.166960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:05.165258 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:05.254960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:05.254047 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.261080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:05.261016 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.302982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:05.294479 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.305089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:05.303275 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.455667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:05.397070 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:05.455667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:05.397531 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:30:05.455667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:05.398097 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:30:05.455667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:05.411495 4883 trace.go:236] Trace[1777527431]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-64bcb8d955-zr5b9" (30-Oct-2026 18:30:04.077) (total time: 1333ms): Oct 30 18:30:05.455667 ip-10-0-130-250 kubenswrapper[4883]: Trace[1777527431]: [1.333662273s] [1.333662273s] END Oct 30 18:30:05.916574 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:05.913389 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:30:06.005985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.003975 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:06.005985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.004298 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:06.005985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.005088 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:30:06.069193 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.069142 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:06.186184 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.174027 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:06.215249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.174195 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:30:06.274508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.274474 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:30:06.495954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.492202 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:06.495954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.493886 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:30:06.696972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.674101 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 30 18:30:06.696972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.679711 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj"] Oct 30 18:30:06.700962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.698321 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:30:06.873295 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.873244 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:30:06.874197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.874167 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:30:06.990145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:06.972260 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded" Oct 30 18:30:07.024010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.013005 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:07.064023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.035084 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:30:07.064023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.035488 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:30:07.097083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.087619 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:07.119158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.118139 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:07.187977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.181051 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:07.540016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.504547 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:07.540016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.504982 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:30:07.720594 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.720541 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" event={"ID":"1041e24a-32c1-49ff-9c78-a1708404d7fd","Type":"ContainerStarted","Data":"e1a9d13f21fc19d1ea3ac9b3235df89f1b4b36db66699e3ab8f1ed4c8f4db385"} Oct 30 18:30:07.721156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.721135 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:30:07.886977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.872184 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:30:07.886977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.872774 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:30:07.915053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.891026 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:07.915053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.891430 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:30:07.948706 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.948666 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" event={"ID":"0213bbf6-2490-46de-a522-f12d2f4f2515","Type":"ContainerStarted","Data":"b23fbe3b3bbf43bce2f30831fc9330f3ddb40c153d388cba1854dae19dea6e4f"} Oct 30 18:30:07.979038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.979001 4883 scope.go:122] "RemoveContainer" containerID="7c85951fed9831de3043bd8137cc89d772a3c9bb995d881f7bf1a92ea0629a65" Oct 30 18:30:07.979788 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:07.979738 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" event={"ID":"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607","Type":"ContainerStarted","Data":"7c85951fed9831de3043bd8137cc89d772a3c9bb995d881f7bf1a92ea0629a65"} Oct 30 18:30:08.072695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.072099 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.104065 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.093192 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:30:08.182175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.182109 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.426960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.408855 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.452369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.448094 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.476081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.453680 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:30:08.485059 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.415098 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:08.485059 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.476638 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.533960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.496045 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded" Oct 30 18:30:08.533960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.496511 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:30:08.533960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.498321 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:30:08.533960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.498452 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.533960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.530197 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:30:08.533960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.530651 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:30:08.961162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.961121 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:30:08.986622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.986582 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:08.994722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:08.994665 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.184172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.184124 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.251647 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.229871 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Oct 30 18:30:09.251910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.229908 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:30:09.295433 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.295394 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:30:09.474101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.474055 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.535594 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.487537 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 30 18:30:09.535594 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.488860 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.566902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.536018 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.783429 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.783379 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.811312 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.791652 4883 generic.go:356] "Generic (PLEG): container finished" podID="0213bbf6-2490-46de-a522-f12d2f4f2515" containerID="675a61ad0be6c0874bc1e12773c2e6d3d9db0f79f06be27d29add12a93f6300f" exitCode=60 Oct 30 18:30:09.811632 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.792077 4883 scope.go:122] "RemoveContainer" containerID="675a61ad0be6c0874bc1e12773c2e6d3d9db0f79f06be27d29add12a93f6300f" Oct 30 18:30:09.902958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.883448 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:30:09.902958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.888871 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Oct 30 18:30:09.910920 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.631810 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.912645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.560658 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:09.917021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.916956 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:30:09.917021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.917002 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" event={"ID":"0213bbf6-2490-46de-a522-f12d2f4f2515","Type":"ContainerDied","Data":"675a61ad0be6c0874bc1e12773c2e6d3d9db0f79f06be27d29add12a93f6300f"} Oct 30 18:30:09.917245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.917073 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:30:09.917245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.917108 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:30:09.917245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.917138 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:30:09.917245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.917166 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:30:09.917245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.917193 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:30:09.977986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.977956 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:30:09.987235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.987199 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" event={"ID":"96a19637-c44e-414a-b3b2-d963cf2ec937","Type":"ContainerStarted","Data":"c65b9c6ee1534e908ccf4a981ad8915422872f040b95ce055b6ecc908d196a8b"} Oct 30 18:30:09.987377 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.987326 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:30:09.987377 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:09.987369 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:30:10.016004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.013786 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:30:10.025623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.021568 4883 generic.go:356] "Generic (PLEG): container finished" podID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" containerID="7c85951fed9831de3043bd8137cc89d772a3c9bb995d881f7bf1a92ea0629a65" exitCode=60 Oct 30 18:30:10.025623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.021694 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" event={"ID":"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607","Type":"ContainerDied","Data":"7c85951fed9831de3043bd8137cc89d772a3c9bb995d881f7bf1a92ea0629a65"} Oct 30 18:30:10.028486 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.028282 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:30:10.029072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.029048 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx"] Oct 30 18:30:10.030293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.030266 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" event={"ID":"77be19c9-c897-46ac-bcfc-da3e3623e2f5","Type":"ContainerStarted","Data":"502e7f35dddae5e3cebd90725ddd4e45901f216a48ed50f554e52691ae7a8100"} Oct 30 18:30:10.033134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.033108 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" event={"ID":"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e","Type":"ContainerStarted","Data":"94ad5d6556381247760043da1f9c53be3c76173f20f67f0d6b11cddd913f335b"} Oct 30 18:30:10.036323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.035953 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz"] Oct 30 18:30:10.043957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.043920 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj"] Oct 30 18:30:10.050869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.050827 4883 scope.go:122] "RemoveContainer" containerID="4d18526e75fa29431f1237b58d5e108d88fa203ad057f05d98e8d37bc0732b61" Oct 30 18:30:10.060983 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:10.052642 4883 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0213bbf6_2490_46de_a522_f12d2f4f2515.slice/crio-675a61ad0be6c0874bc1e12773c2e6d3d9db0f79f06be27d29add12a93f6300f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0213bbf6_2490_46de_a522_f12d2f4f2515.slice/crio-conmon-675a61ad0be6c0874bc1e12773c2e6d3d9db0f79f06be27d29add12a93f6300f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:30:10.067867 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.063388 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" event={"ID":"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8","Type":"ContainerStarted","Data":"7dda001f53bebb14337cafa48c03a9d4122cf0bab98f40696913c79fc93f46b2"} Oct 30 18:30:10.078318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.075632 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h"] Oct 30 18:30:10.082860 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.082833 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj"] Oct 30 18:30:10.095068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.095035 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8"] Oct 30 18:30:10.126180 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.126143 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" event={"ID":"888e9299-97b3-4cef-9213-69663ac795b0","Type":"ContainerStarted","Data":"19396b49c7eb8c5ddd57446ac3a214c89c1f251939e4a1607ec341ebb56951e4"} Oct 30 18:30:10.137239 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.137207 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7"] Oct 30 18:30:10.290093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.289981 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" podStartSLOduration=10.289924213 podStartE2EDuration="10.289924213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:10.288458599 +0000 UTC m=+2269.245462436" watchObservedRunningTime="2026-10-30 18:30:10.289924213 +0000 UTC m=+2269.246928051" Oct 30 18:30:10.419466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:10.419415 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" podStartSLOduration=10.419400176 podStartE2EDuration="10.419400176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:10.418064055 +0000 UTC m=+2269.375067902" watchObservedRunningTime="2026-10-30 18:30:10.419400176 +0000 UTC m=+2269.376404057" Oct 30 18:30:11.135640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.135551 4883 generic.go:356] "Generic (PLEG): container finished" podID="96a19637-c44e-414a-b3b2-d963cf2ec937" containerID="7e30de6dfe1df39663087b6f1ad91ec8ee0a15dc40f42a4cbbcd91bbd9664a21" exitCode=60 Oct 30 18:30:11.136088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.135634 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" event={"ID":"96a19637-c44e-414a-b3b2-d963cf2ec937","Type":"ContainerDied","Data":"7e30de6dfe1df39663087b6f1ad91ec8ee0a15dc40f42a4cbbcd91bbd9664a21"} Oct 30 18:30:11.136088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.136067 4883 scope.go:122] "RemoveContainer" containerID="7e30de6dfe1df39663087b6f1ad91ec8ee0a15dc40f42a4cbbcd91bbd9664a21" Oct 30 18:30:11.138327 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.138300 4883 generic.go:356] "Generic (PLEG): container finished" podID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" containerID="800c6876bfc732821d26ee70c36ee408852c40e08692cacd8a5491d3946479d1" exitCode=60 Oct 30 18:30:11.138441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.138422 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" event={"ID":"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607","Type":"ContainerDied","Data":"800c6876bfc732821d26ee70c36ee408852c40e08692cacd8a5491d3946479d1"} Oct 30 18:30:11.138576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.138478 4883 scope.go:122] "RemoveContainer" containerID="7c85951fed9831de3043bd8137cc89d772a3c9bb995d881f7bf1a92ea0629a65" Oct 30 18:30:11.138846 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.138824 4883 scope.go:122] "RemoveContainer" containerID="800c6876bfc732821d26ee70c36ee408852c40e08692cacd8a5491d3946479d1" Oct 30 18:30:11.140113 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:11.139697 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj_odh-ai-gateway-infra(2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" Oct 30 18:30:11.140973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.140450 4883 generic.go:356] "Generic (PLEG): container finished" podID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" containerID="dc79473a859fbaf4a026020493f40e63d71a3cdf850dc7631d7eac05fd22fbfb" exitCode=60 Oct 30 18:30:11.140973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.140565 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" event={"ID":"77be19c9-c897-46ac-bcfc-da3e3623e2f5","Type":"ContainerDied","Data":"dc79473a859fbaf4a026020493f40e63d71a3cdf850dc7631d7eac05fd22fbfb"} Oct 30 18:30:11.140973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.140751 4883 scope.go:122] "RemoveContainer" containerID="dc79473a859fbaf4a026020493f40e63d71a3cdf850dc7631d7eac05fd22fbfb" Oct 30 18:30:11.142650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.142627 4883 generic.go:356] "Generic (PLEG): container finished" podID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" containerID="ffedb2d96bf70a8a082079101b2de6c0d72d47cee9b8e70c6cd652c6db40ed14" exitCode=60 Oct 30 18:30:11.142766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.142730 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" event={"ID":"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e","Type":"ContainerDied","Data":"ffedb2d96bf70a8a082079101b2de6c0d72d47cee9b8e70c6cd652c6db40ed14"} Oct 30 18:30:11.142982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.142953 4883 scope.go:122] "RemoveContainer" containerID="ffedb2d96bf70a8a082079101b2de6c0d72d47cee9b8e70c6cd652c6db40ed14" Oct 30 18:30:11.144922 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.144903 4883 generic.go:356] "Generic (PLEG): container finished" podID="1041e24a-32c1-49ff-9c78-a1708404d7fd" containerID="4d18526e75fa29431f1237b58d5e108d88fa203ad057f05d98e8d37bc0732b61" exitCode=60 Oct 30 18:30:11.145053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.144962 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" event={"ID":"1041e24a-32c1-49ff-9c78-a1708404d7fd","Type":"ContainerDied","Data":"4d18526e75fa29431f1237b58d5e108d88fa203ad057f05d98e8d37bc0732b61"} Oct 30 18:30:11.145053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.145007 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" event={"ID":"1041e24a-32c1-49ff-9c78-a1708404d7fd","Type":"ContainerStarted","Data":"cce65fc97069243b3710fd8958366be00f844ebf8ba8d7a232ba9c916b74af3e"} Oct 30 18:30:11.146648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.146625 4883 generic.go:356] "Generic (PLEG): container finished" podID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" containerID="0c8a05f75a92aca162b7e3f0e3777c1a47436d4f3930a1a1d94698c929558144" exitCode=60 Oct 30 18:30:11.146756 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.146689 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" event={"ID":"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8","Type":"ContainerDied","Data":"0c8a05f75a92aca162b7e3f0e3777c1a47436d4f3930a1a1d94698c929558144"} Oct 30 18:30:11.146906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.146890 4883 scope.go:122] "RemoveContainer" containerID="0c8a05f75a92aca162b7e3f0e3777c1a47436d4f3930a1a1d94698c929558144" Oct 30 18:30:11.148432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.148326 4883 generic.go:356] "Generic (PLEG): container finished" podID="888e9299-97b3-4cef-9213-69663ac795b0" containerID="5c85297a39282f5b599e0ea62281ddb6c1e379d1b586ea07c94a2819c4aeb6f9" exitCode=60 Oct 30 18:30:11.148513 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.148480 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" event={"ID":"888e9299-97b3-4cef-9213-69663ac795b0","Type":"ContainerDied","Data":"5c85297a39282f5b599e0ea62281ddb6c1e379d1b586ea07c94a2819c4aeb6f9"} Oct 30 18:30:11.149099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.148583 4883 scope.go:122] "RemoveContainer" containerID="5c85297a39282f5b599e0ea62281ddb6c1e379d1b586ea07c94a2819c4aeb6f9" Oct 30 18:30:11.155100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.155080 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" event={"ID":"0213bbf6-2490-46de-a522-f12d2f4f2515","Type":"ContainerStarted","Data":"7b72c26780cec0c5f0a6372d2a43b1f8f4306effbe8d1f01b993d2c2d4b38f7e"} Oct 30 18:30:11.318088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.318034 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" podStartSLOduration=11.318019838 podStartE2EDuration="11.318019838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:11.313603017 +0000 UTC m=+2270.270606852" watchObservedRunningTime="2026-10-30 18:30:11.318019838 +0000 UTC m=+2270.275023720" Oct 30 18:30:11.401258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:11.401212 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" podStartSLOduration=11.401198932 podStartE2EDuration="11.401198932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:11.396898466 +0000 UTC m=+2270.353902283" watchObservedRunningTime="2026-10-30 18:30:11.401198932 +0000 UTC m=+2270.358202812" Oct 30 18:30:12.168101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.167622 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" event={"ID":"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8","Type":"ContainerStarted","Data":"33c01c1a2ec2ccc9da0abb493051fe980a55d77b8391365fe7d82812292226cd"} Oct 30 18:30:12.171746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.171705 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" event={"ID":"888e9299-97b3-4cef-9213-69663ac795b0","Type":"ContainerStarted","Data":"a8afd91457abfb341d678748cf523276bc86549bd407a9dcb03d10bd7bbdf18c"} Oct 30 18:30:12.174743 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.174714 4883 generic.go:356] "Generic (PLEG): container finished" podID="0213bbf6-2490-46de-a522-f12d2f4f2515" containerID="7b72c26780cec0c5f0a6372d2a43b1f8f4306effbe8d1f01b993d2c2d4b38f7e" exitCode=60 Oct 30 18:30:12.174911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.174793 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" event={"ID":"0213bbf6-2490-46de-a522-f12d2f4f2515","Type":"ContainerDied","Data":"7b72c26780cec0c5f0a6372d2a43b1f8f4306effbe8d1f01b993d2c2d4b38f7e"} Oct 30 18:30:12.174911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.174850 4883 scope.go:122] "RemoveContainer" containerID="675a61ad0be6c0874bc1e12773c2e6d3d9db0f79f06be27d29add12a93f6300f" Oct 30 18:30:12.175074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.174975 4883 scope.go:122] "RemoveContainer" containerID="7b72c26780cec0c5f0a6372d2a43b1f8f4306effbe8d1f01b993d2c2d4b38f7e" Oct 30 18:30:12.175292 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:12.175271 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz_odh-ai-gateway-infra(0213bbf6-2490-46de-a522-f12d2f4f2515)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" Oct 30 18:30:12.185313 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.185280 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" event={"ID":"96a19637-c44e-414a-b3b2-d963cf2ec937","Type":"ContainerStarted","Data":"138b1ee68d20f8a290bd1751c4b4b52ab415c92ba0f66ee49ef1b9bae857cca6"} Oct 30 18:30:12.187592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.187568 4883 scope.go:122] "RemoveContainer" containerID="800c6876bfc732821d26ee70c36ee408852c40e08692cacd8a5491d3946479d1" Oct 30 18:30:12.187926 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:12.187900 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj_odh-ai-gateway-infra(2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" Oct 30 18:30:12.190135 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.190104 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" event={"ID":"77be19c9-c897-46ac-bcfc-da3e3623e2f5","Type":"ContainerStarted","Data":"f0368dc7724f791acbfd3045ab51e74e8a6e4779144907334e4a999d5c226477"} Oct 30 18:30:12.211679 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.193233 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" event={"ID":"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e","Type":"ContainerStarted","Data":"739fb233a79e82c3ed324c00bae6e3bea2b1a5b36d70e3f4d7a258658c281d54"} Oct 30 18:30:12.211679 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.195922 4883 generic.go:356] "Generic (PLEG): container finished" podID="1041e24a-32c1-49ff-9c78-a1708404d7fd" containerID="cce65fc97069243b3710fd8958366be00f844ebf8ba8d7a232ba9c916b74af3e" exitCode=60 Oct 30 18:30:12.211679 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.196022 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" event={"ID":"1041e24a-32c1-49ff-9c78-a1708404d7fd","Type":"ContainerDied","Data":"cce65fc97069243b3710fd8958366be00f844ebf8ba8d7a232ba9c916b74af3e"} Oct 30 18:30:12.211679 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.196397 4883 scope.go:122] "RemoveContainer" containerID="cce65fc97069243b3710fd8958366be00f844ebf8ba8d7a232ba9c916b74af3e" Oct 30 18:30:12.211679 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:12.196697 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx_odh-ai-gateway-infra(1041e24a-32c1-49ff-9c78-a1708404d7fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" Oct 30 18:30:12.211679 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.200984 4883 scope.go:122] "RemoveContainer" containerID="4d18526e75fa29431f1237b58d5e108d88fa203ad057f05d98e8d37bc0732b61" Oct 30 18:30:12.215116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.215058 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" podStartSLOduration=12.215039351 podStartE2EDuration="12.215039351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:12.211573523 +0000 UTC m=+2271.168577361" watchObservedRunningTime="2026-10-30 18:30:12.215039351 +0000 UTC m=+2271.172043188" Oct 30 18:30:12.256064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.256012 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" podStartSLOduration=12.255997854 podStartE2EDuration="12.255997854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:12.254513862 +0000 UTC m=+2271.211517700" watchObservedRunningTime="2026-10-30 18:30:12.255997854 +0000 UTC m=+2271.213001691" Oct 30 18:30:12.427117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:12.427012 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" podStartSLOduration=12.426998225 podStartE2EDuration="12.426998225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:12.424112726 +0000 UTC m=+2271.381116613" watchObservedRunningTime="2026-10-30 18:30:12.426998225 +0000 UTC m=+2271.384002061" Oct 30 18:30:13.206419 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.206390 4883 scope.go:122] "RemoveContainer" containerID="7b72c26780cec0c5f0a6372d2a43b1f8f4306effbe8d1f01b993d2c2d4b38f7e" Oct 30 18:30:13.206905 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:13.206880 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz_odh-ai-gateway-infra(0213bbf6-2490-46de-a522-f12d2f4f2515)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" Oct 30 18:30:13.208246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.208221 4883 generic.go:356] "Generic (PLEG): container finished" podID="96a19637-c44e-414a-b3b2-d963cf2ec937" containerID="138b1ee68d20f8a290bd1751c4b4b52ab415c92ba0f66ee49ef1b9bae857cca6" exitCode=60 Oct 30 18:30:13.208347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.208299 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" event={"ID":"96a19637-c44e-414a-b3b2-d963cf2ec937","Type":"ContainerDied","Data":"138b1ee68d20f8a290bd1751c4b4b52ab415c92ba0f66ee49ef1b9bae857cca6"} Oct 30 18:30:13.208388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.208356 4883 scope.go:122] "RemoveContainer" containerID="7e30de6dfe1df39663087b6f1ad91ec8ee0a15dc40f42a4cbbcd91bbd9664a21" Oct 30 18:30:13.208529 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.208511 4883 scope.go:122] "RemoveContainer" containerID="138b1ee68d20f8a290bd1751c4b4b52ab415c92ba0f66ee49ef1b9bae857cca6" Oct 30 18:30:13.208849 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:13.208825 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8_odh-ai-gateway-infra(96a19637-c44e-414a-b3b2-d963cf2ec937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" Oct 30 18:30:13.210265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.210239 4883 generic.go:356] "Generic (PLEG): container finished" podID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" containerID="f0368dc7724f791acbfd3045ab51e74e8a6e4779144907334e4a999d5c226477" exitCode=60 Oct 30 18:30:13.210340 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.210274 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" event={"ID":"77be19c9-c897-46ac-bcfc-da3e3623e2f5","Type":"ContainerDied","Data":"f0368dc7724f791acbfd3045ab51e74e8a6e4779144907334e4a999d5c226477"} Oct 30 18:30:13.210536 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.210523 4883 scope.go:122] "RemoveContainer" containerID="f0368dc7724f791acbfd3045ab51e74e8a6e4779144907334e4a999d5c226477" Oct 30 18:30:13.210789 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:13.210771 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h_odh-ai-gateway-infra(77be19c9-c897-46ac-bcfc-da3e3623e2f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" Oct 30 18:30:13.212167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.212146 4883 generic.go:356] "Generic (PLEG): container finished" podID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" containerID="739fb233a79e82c3ed324c00bae6e3bea2b1a5b36d70e3f4d7a258658c281d54" exitCode=60 Oct 30 18:30:13.212303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.212221 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" event={"ID":"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e","Type":"ContainerDied","Data":"739fb233a79e82c3ed324c00bae6e3bea2b1a5b36d70e3f4d7a258658c281d54"} Oct 30 18:30:13.212412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.212399 4883 scope.go:122] "RemoveContainer" containerID="739fb233a79e82c3ed324c00bae6e3bea2b1a5b36d70e3f4d7a258658c281d54" Oct 30 18:30:13.212664 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:13.212631 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj_odh-ai-gateway-infra(add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" Oct 30 18:30:13.214201 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.214187 4883 scope.go:122] "RemoveContainer" containerID="cce65fc97069243b3710fd8958366be00f844ebf8ba8d7a232ba9c916b74af3e" Oct 30 18:30:13.214430 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:13.214407 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx_odh-ai-gateway-infra(1041e24a-32c1-49ff-9c78-a1708404d7fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" Oct 30 18:30:13.215784 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.215763 4883 generic.go:356] "Generic (PLEG): container finished" podID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" containerID="33c01c1a2ec2ccc9da0abb493051fe980a55d77b8391365fe7d82812292226cd" exitCode=60 Oct 30 18:30:13.215870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.215841 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" event={"ID":"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8","Type":"ContainerDied","Data":"33c01c1a2ec2ccc9da0abb493051fe980a55d77b8391365fe7d82812292226cd"} Oct 30 18:30:13.216061 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.216044 4883 scope.go:122] "RemoveContainer" containerID="33c01c1a2ec2ccc9da0abb493051fe980a55d77b8391365fe7d82812292226cd" Oct 30 18:30:13.216305 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:13.216284 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7_odh-ai-gateway-infra(8ac12c7a-1620-439a-93f1-eb8ac4a04aa8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" Oct 30 18:30:13.217829 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.217799 4883 generic.go:356] "Generic (PLEG): container finished" podID="888e9299-97b3-4cef-9213-69663ac795b0" containerID="a8afd91457abfb341d678748cf523276bc86549bd407a9dcb03d10bd7bbdf18c" exitCode=60 Oct 30 18:30:13.217922 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.217820 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" event={"ID":"888e9299-97b3-4cef-9213-69663ac795b0","Type":"ContainerDied","Data":"a8afd91457abfb341d678748cf523276bc86549bd407a9dcb03d10bd7bbdf18c"} Oct 30 18:30:13.218060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.218043 4883 scope.go:122] "RemoveContainer" containerID="a8afd91457abfb341d678748cf523276bc86549bd407a9dcb03d10bd7bbdf18c" Oct 30 18:30:13.218272 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:13.218254 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj_odh-ai-gateway-infra(888e9299-97b3-4cef-9213-69663ac795b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" podUID="888e9299-97b3-4cef-9213-69663ac795b0" Oct 30 18:30:13.222586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.222568 4883 scope.go:122] "RemoveContainer" containerID="dc79473a859fbaf4a026020493f40e63d71a3cdf850dc7631d7eac05fd22fbfb" Oct 30 18:30:13.236353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.236333 4883 scope.go:122] "RemoveContainer" containerID="ffedb2d96bf70a8a082079101b2de6c0d72d47cee9b8e70c6cd652c6db40ed14" Oct 30 18:30:13.250471 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.250452 4883 scope.go:122] "RemoveContainer" containerID="0c8a05f75a92aca162b7e3f0e3777c1a47436d4f3930a1a1d94698c929558144" Oct 30 18:30:13.269707 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:13.269680 4883 scope.go:122] "RemoveContainer" containerID="5c85297a39282f5b599e0ea62281ddb6c1e379d1b586ea07c94a2819c4aeb6f9" Oct 30 18:30:14.229350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.229069 4883 scope.go:122] "RemoveContainer" containerID="f0368dc7724f791acbfd3045ab51e74e8a6e4779144907334e4a999d5c226477" Oct 30 18:30:14.229827 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:14.229493 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h_odh-ai-gateway-infra(77be19c9-c897-46ac-bcfc-da3e3623e2f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" Oct 30 18:30:14.231405 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.231378 4883 scope.go:122] "RemoveContainer" containerID="739fb233a79e82c3ed324c00bae6e3bea2b1a5b36d70e3f4d7a258658c281d54" Oct 30 18:30:14.231724 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:14.231699 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj_odh-ai-gateway-infra(add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" Oct 30 18:30:14.233449 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.233432 4883 scope.go:122] "RemoveContainer" containerID="33c01c1a2ec2ccc9da0abb493051fe980a55d77b8391365fe7d82812292226cd" Oct 30 18:30:14.233716 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:14.233698 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7_odh-ai-gateway-infra(8ac12c7a-1620-439a-93f1-eb8ac4a04aa8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" Oct 30 18:30:14.235045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.235029 4883 scope.go:122] "RemoveContainer" containerID="a8afd91457abfb341d678748cf523276bc86549bd407a9dcb03d10bd7bbdf18c" Oct 30 18:30:14.235229 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:14.235209 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj_odh-ai-gateway-infra(888e9299-97b3-4cef-9213-69663ac795b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" podUID="888e9299-97b3-4cef-9213-69663ac795b0" Oct 30 18:30:14.236772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.236754 4883 scope.go:122] "RemoveContainer" containerID="138b1ee68d20f8a290bd1751c4b4b52ab415c92ba0f66ee49ef1b9bae857cca6" Oct 30 18:30:14.236976 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:14.236954 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8_odh-ai-gateway-infra(96a19637-c44e-414a-b3b2-d963cf2ec937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" Oct 30 18:30:14.388025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.382208 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd"] Oct 30 18:30:14.411487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.409413 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.412656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.412586 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd"] Oct 30 18:30:14.481043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.477286 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c8eef3-openshift-default-dockercfg-4hdd4\"" Oct 30 18:30:14.543585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.530737 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-socket\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.543585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.530788 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istiod-ca-cert\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.543585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.530849 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-podinfo\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.543585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.530926 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-certs\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.543585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.531051 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-token\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.543585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.531100 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-envoy\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.543585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.531235 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-credential-socket\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.543585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.531290 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2nv7\" (UniqueName: \"kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-kube-api-access-x2nv7\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.543585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.531344 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-data\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.632503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.632464 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-credential-socket\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.632687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.632519 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2nv7\" (UniqueName: \"kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-kube-api-access-x2nv7\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.632687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.632545 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-data\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.632687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.632603 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-socket\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.632687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.632623 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istiod-ca-cert\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.632687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.632681 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-podinfo\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.633010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.632761 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-certs\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.633010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.632813 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-token\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.633010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.632863 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-envoy\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.633154 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.633109 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-credential-socket\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.633154 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.633136 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-data\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.633372 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.633348 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-certs\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.633470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.633407 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-socket\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.634047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.634021 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istiod-ca-cert\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.635821 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.635774 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-podinfo\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.635915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.635837 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-envoy\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.661161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.661120 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2nv7\" (UniqueName: \"kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-kube-api-access-x2nv7\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.663683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.663015 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-token\") pod \"e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:14.744414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:14.744310 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:15.085180 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:15.085149 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd"] Oct 30 18:30:15.087310 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:30:15.087283 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5af977_f5ee_480f_bcc1_8c4a5ff5257d.slice/crio-6c4bfb77bab844335cf116a94708f9470ca53626cd363b39c31f2e96e4461954 WatchSource:0}: Error finding container 6c4bfb77bab844335cf116a94708f9470ca53626cd363b39c31f2e96e4461954: Status 404 returned error can't find the container with id 6c4bfb77bab844335cf116a94708f9470ca53626cd363b39c31f2e96e4461954 Oct 30 18:30:15.090694 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:15.090659 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:30:15.090768 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:15.090742 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:30:15.090812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:15.090774 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:30:15.252955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:15.252020 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" event={"ID":"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d","Type":"ContainerStarted","Data":"42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3"} Oct 30 18:30:15.252955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:15.252090 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" event={"ID":"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d","Type":"ContainerStarted","Data":"6c4bfb77bab844335cf116a94708f9470ca53626cd363b39c31f2e96e4461954"} Oct 30 18:30:15.310385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:15.309652 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" podStartSLOduration=1.309631245 podStartE2EDuration="1.309631245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:15.300109185 +0000 UTC m=+2274.257113034" watchObservedRunningTime="2026-10-30 18:30:15.309631245 +0000 UTC m=+2274.266635082" Oct 30 18:30:15.744616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:15.744575 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:16.749975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:16.749946 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:17.270173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:17.270138 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:17.271329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:17.271303 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:30:20.382063 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.382015 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8"] Oct 30 18:30:20.386680 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.386650 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.402794 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.402770 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a2a10a-openshift-default-dockercfg-4fw27\"" Oct 30 18:30:20.422159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.422132 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8"] Oct 30 18:30:20.526454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.526414 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f24af4-568d-4e91-8458-79fad2821ad3-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.526454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.526452 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f24af4-568d-4e91-8458-79fad2821ad3-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.526701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.526546 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.526701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.526654 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwlbf\" (UniqueName: \"kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-kube-api-access-qwlbf\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.526834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.526716 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.526834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.526756 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.526834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.526800 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.526993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.526958 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.526993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.526985 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.628383 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.628344 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.628383 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.628385 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.628644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.628430 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.628644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.628564 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.628644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.628607 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.628807 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.628711 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f24af4-568d-4e91-8458-79fad2821ad3-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.628807 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.628741 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f24af4-568d-4e91-8458-79fad2821ad3-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.628807 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.628781 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.628995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.628835 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwlbf\" (UniqueName: \"kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-kube-api-access-qwlbf\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.629540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.629494 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.629658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.629633 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.634068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.630100 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.634068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.630404 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.634068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.630837 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f24af4-568d-4e91-8458-79fad2821ad3-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.634068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.631223 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.645045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.645014 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f24af4-568d-4e91-8458-79fad2821ad3-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.652837 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.652809 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.654381 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.654355 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwlbf\" (UniqueName: \"kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-kube-api-access-qwlbf\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:20.702318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:20.702283 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:21.028972 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:30:21.028925 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f24af4_568d_4e91_8458_79fad2821ad3.slice/crio-21e0f0b7d8ddd1131f517208e544fb25d986da4c6c8c306259b5f5a49248b8ad WatchSource:0}: Error finding container 21e0f0b7d8ddd1131f517208e544fb25d986da4c6c8c306259b5f5a49248b8ad: Status 404 returned error can't find the container with id 21e0f0b7d8ddd1131f517208e544fb25d986da4c6c8c306259b5f5a49248b8ad Oct 30 18:30:21.029068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:21.029007 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8"] Oct 30 18:30:21.031583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:21.031553 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:30:21.031639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:21.031617 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:30:21.031672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:21.031647 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:30:21.310069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:21.309973 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" event={"ID":"56f24af4-568d-4e91-8458-79fad2821ad3","Type":"ContainerStarted","Data":"db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75"} Oct 30 18:30:21.310069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:21.310031 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" event={"ID":"56f24af4-568d-4e91-8458-79fad2821ad3","Type":"ContainerStarted","Data":"21e0f0b7d8ddd1131f517208e544fb25d986da4c6c8c306259b5f5a49248b8ad"} Oct 30 18:30:21.359037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:21.358986 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" podStartSLOduration=1.358973092 podStartE2EDuration="1.358973092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:21.357195953 +0000 UTC m=+2280.314199789" watchObservedRunningTime="2026-10-30 18:30:21.358973092 +0000 UTC m=+2280.315976927" Oct 30 18:30:21.702592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:21.702561 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:22.703974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:22.703929 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:30:22.704374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:22.703994 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" podUID="56f24af4-568d-4e91-8458-79fad2821ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 18:30:23.186959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:23.185373 4883 scope.go:122] "RemoveContainer" containerID="cce65fc97069243b3710fd8958366be00f844ebf8ba8d7a232ba9c916b74af3e" Oct 30 18:30:23.186959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:23.186272 4883 scope.go:122] "RemoveContainer" containerID="7b72c26780cec0c5f0a6372d2a43b1f8f4306effbe8d1f01b993d2c2d4b38f7e" Oct 30 18:30:23.186959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:23.186951 4883 scope.go:122] "RemoveContainer" containerID="f0368dc7724f791acbfd3045ab51e74e8a6e4779144907334e4a999d5c226477" Oct 30 18:30:23.704105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:23.704067 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:30:23.704464 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:23.704127 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" podUID="56f24af4-568d-4e91-8458-79fad2821ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 18:30:24.172748 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:24.172711 4883 scope.go:122] "RemoveContainer" containerID="a8afd91457abfb341d678748cf523276bc86549bd407a9dcb03d10bd7bbdf18c" Oct 30 18:30:24.336172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:24.336139 4883 generic.go:356] "Generic (PLEG): container finished" podID="0213bbf6-2490-46de-a522-f12d2f4f2515" containerID="bbed34cfdf6d12d4785e45d743e36de0390858a9fc77bf94acdad8f25abec019" exitCode=60 Oct 30 18:30:24.336344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:24.336209 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" event={"ID":"0213bbf6-2490-46de-a522-f12d2f4f2515","Type":"ContainerDied","Data":"bbed34cfdf6d12d4785e45d743e36de0390858a9fc77bf94acdad8f25abec019"} Oct 30 18:30:24.336344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:24.336265 4883 scope.go:122] "RemoveContainer" containerID="7b72c26780cec0c5f0a6372d2a43b1f8f4306effbe8d1f01b993d2c2d4b38f7e" Oct 30 18:30:24.337119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:24.336731 4883 scope.go:122] "RemoveContainer" containerID="bbed34cfdf6d12d4785e45d743e36de0390858a9fc77bf94acdad8f25abec019" Oct 30 18:30:24.337210 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:24.337193 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz_odh-ai-gateway-infra(0213bbf6-2490-46de-a522-f12d2f4f2515)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" Oct 30 18:30:24.338566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:24.338505 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" event={"ID":"77be19c9-c897-46ac-bcfc-da3e3623e2f5","Type":"ContainerStarted","Data":"cc3e853ca9640c10056a90e9d53bdfc8be8bfae0af7c3371b57e93fec3e9ab2b"} Oct 30 18:30:24.340546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:24.340520 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" event={"ID":"1041e24a-32c1-49ff-9c78-a1708404d7fd","Type":"ContainerStarted","Data":"06ec51b19897c9e588cc57aff6811c7f55b84b50ecb4b7c9782cbb17f7dd9396"} Oct 30 18:30:24.703251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:24.703209 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:30:24.703388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:24.703266 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" podUID="56f24af4-568d-4e91-8458-79fad2821ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 18:30:25.182583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.182527 4883 scope.go:122] "RemoveContainer" containerID="138b1ee68d20f8a290bd1751c4b4b52ab415c92ba0f66ee49ef1b9bae857cca6" Oct 30 18:30:25.183508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.183483 4883 scope.go:122] "RemoveContainer" containerID="800c6876bfc732821d26ee70c36ee408852c40e08692cacd8a5491d3946479d1" Oct 30 18:30:25.356345 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.356311 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" event={"ID":"888e9299-97b3-4cef-9213-69663ac795b0","Type":"ContainerStarted","Data":"800fbffbaf5055155e5e6095df9a70a837d2fd886ff02e555ac356b8eba55ee1"} Oct 30 18:30:25.360461 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.360437 4883 generic.go:356] "Generic (PLEG): container finished" podID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" containerID="cc3e853ca9640c10056a90e9d53bdfc8be8bfae0af7c3371b57e93fec3e9ab2b" exitCode=60 Oct 30 18:30:25.360746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.360542 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" event={"ID":"77be19c9-c897-46ac-bcfc-da3e3623e2f5","Type":"ContainerDied","Data":"cc3e853ca9640c10056a90e9d53bdfc8be8bfae0af7c3371b57e93fec3e9ab2b"} Oct 30 18:30:25.360828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.360796 4883 scope.go:122] "RemoveContainer" containerID="f0368dc7724f791acbfd3045ab51e74e8a6e4779144907334e4a999d5c226477" Oct 30 18:30:25.361260 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.361239 4883 scope.go:122] "RemoveContainer" containerID="cc3e853ca9640c10056a90e9d53bdfc8be8bfae0af7c3371b57e93fec3e9ab2b" Oct 30 18:30:25.361576 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:25.361557 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h_odh-ai-gateway-infra(77be19c9-c897-46ac-bcfc-da3e3623e2f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" Oct 30 18:30:25.367961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.364463 4883 generic.go:356] "Generic (PLEG): container finished" podID="1041e24a-32c1-49ff-9c78-a1708404d7fd" containerID="06ec51b19897c9e588cc57aff6811c7f55b84b50ecb4b7c9782cbb17f7dd9396" exitCode=60 Oct 30 18:30:25.367961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.364531 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" event={"ID":"1041e24a-32c1-49ff-9c78-a1708404d7fd","Type":"ContainerDied","Data":"06ec51b19897c9e588cc57aff6811c7f55b84b50ecb4b7c9782cbb17f7dd9396"} Oct 30 18:30:25.367961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.364911 4883 scope.go:122] "RemoveContainer" containerID="06ec51b19897c9e588cc57aff6811c7f55b84b50ecb4b7c9782cbb17f7dd9396" Oct 30 18:30:25.367961 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:25.365256 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx_odh-ai-gateway-infra(1041e24a-32c1-49ff-9c78-a1708404d7fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" Oct 30 18:30:25.409965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.409919 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz"] Oct 30 18:30:25.445546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.445517 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h"] Oct 30 18:30:25.450445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.450424 4883 scope.go:122] "RemoveContainer" containerID="cce65fc97069243b3710fd8958366be00f844ebf8ba8d7a232ba9c916b74af3e" Oct 30 18:30:25.521227 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.521196 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx"] Oct 30 18:30:25.642191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.642161 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" Oct 30 18:30:25.703611 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.703571 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:30:25.703735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.703638 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" podUID="56f24af4-568d-4e91-8458-79fad2821ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 18:30:25.741766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.741718 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0213bbf6-2490-46de-a522-f12d2f4f2515-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0213bbf6-2490-46de-a522-f12d2f4f2515-maas-api-service-ca\") pod \"0213bbf6-2490-46de-a522-f12d2f4f2515\" (UID: \"0213bbf6-2490-46de-a522-f12d2f4f2515\") " Oct 30 18:30:25.741986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.741966 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0213bbf6-2490-46de-a522-f12d2f4f2515-kube-api-access-6fgvq\" (UniqueName: \"kubernetes.io/projected/0213bbf6-2490-46de-a522-f12d2f4f2515-kube-api-access-6fgvq\") pod \"0213bbf6-2490-46de-a522-f12d2f4f2515\" (UID: \"0213bbf6-2490-46de-a522-f12d2f4f2515\") " Oct 30 18:30:25.744507 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.744408 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0213bbf6-2490-46de-a522-f12d2f4f2515-maas-api-service-ca" pod "0213bbf6-2490-46de-a522-f12d2f4f2515" (UID: "0213bbf6-2490-46de-a522-f12d2f4f2515"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:25.754314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.747324 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0213bbf6-2490-46de-a522-f12d2f4f2515-kube-api-access-6fgvq" pod "0213bbf6-2490-46de-a522-f12d2f4f2515" (UID: "0213bbf6-2490-46de-a522-f12d2f4f2515"). InnerVolumeSpecName "kube-api-access-6fgvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:25.845027 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.844993 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0213bbf6-2490-46de-a522-f12d2f4f2515-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:25.845027 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:25.845022 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fgvq\" (UniqueName: \"kubernetes.io/projected/0213bbf6-2490-46de-a522-f12d2f4f2515-kube-api-access-6fgvq\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:26.184962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.183234 4883 scope.go:122] "RemoveContainer" containerID="739fb233a79e82c3ed324c00bae6e3bea2b1a5b36d70e3f4d7a258658c281d54" Oct 30 18:30:26.184962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.184253 4883 scope.go:122] "RemoveContainer" containerID="33c01c1a2ec2ccc9da0abb493051fe980a55d77b8391365fe7d82812292226cd" Oct 30 18:30:26.398996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.398964 4883 generic.go:356] "Generic (PLEG): container finished" podID="888e9299-97b3-4cef-9213-69663ac795b0" containerID="800fbffbaf5055155e5e6095df9a70a837d2fd886ff02e555ac356b8eba55ee1" exitCode=60 Oct 30 18:30:26.399139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.399116 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" event={"ID":"888e9299-97b3-4cef-9213-69663ac795b0","Type":"ContainerDied","Data":"800fbffbaf5055155e5e6095df9a70a837d2fd886ff02e555ac356b8eba55ee1"} Oct 30 18:30:26.399190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.399182 4883 scope.go:122] "RemoveContainer" containerID="a8afd91457abfb341d678748cf523276bc86549bd407a9dcb03d10bd7bbdf18c" Oct 30 18:30:26.399574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.399563 4883 scope.go:122] "RemoveContainer" containerID="800fbffbaf5055155e5e6095df9a70a837d2fd886ff02e555ac356b8eba55ee1" Oct 30 18:30:26.399907 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:26.399889 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj_odh-ai-gateway-infra(888e9299-97b3-4cef-9213-69663ac795b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" podUID="888e9299-97b3-4cef-9213-69663ac795b0" Oct 30 18:30:26.401845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.401824 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" event={"ID":"0213bbf6-2490-46de-a522-f12d2f4f2515","Type":"ContainerDied","Data":"b23fbe3b3bbf43bce2f30831fc9330f3ddb40c153d388cba1854dae19dea6e4f"} Oct 30 18:30:26.401920 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.401901 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz" Oct 30 18:30:26.404210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.404190 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" event={"ID":"96a19637-c44e-414a-b3b2-d963cf2ec937","Type":"ContainerStarted","Data":"d79504d2a9478ce60d70c509ba280dcb487c4197c3025dc9684601ed343e5254"} Oct 30 18:30:26.404603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.404587 4883 scope.go:122] "RemoveContainer" containerID="d79504d2a9478ce60d70c509ba280dcb487c4197c3025dc9684601ed343e5254" Oct 30 18:30:26.405102 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:26.405004 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8_odh-ai-gateway-infra(96a19637-c44e-414a-b3b2-d963cf2ec937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" Oct 30 18:30:26.407073 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.407052 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" event={"ID":"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607","Type":"ContainerStarted","Data":"38756e377bb8d58d9f37f4bc3ad8b9a91bf3f5755a36c2a0ffb35e14b4dbae9e"} Oct 30 18:30:26.407441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.407426 4883 scope.go:122] "RemoveContainer" containerID="38756e377bb8d58d9f37f4bc3ad8b9a91bf3f5755a36c2a0ffb35e14b4dbae9e" Oct 30 18:30:26.407963 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:26.407703 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj_odh-ai-gateway-infra(2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" Oct 30 18:30:26.433825 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.433672 4883 scope.go:122] "RemoveContainer" containerID="bbed34cfdf6d12d4785e45d743e36de0390858a9fc77bf94acdad8f25abec019" Oct 30 18:30:26.438659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.437677 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj"] Oct 30 18:30:26.703901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.702985 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:30:26.703901 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.703081 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" podUID="56f24af4-568d-4e91-8458-79fad2821ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 18:30:26.745756 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.745723 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz"] Oct 30 18:30:26.749917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.749890 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" Oct 30 18:30:26.752794 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.752769 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-419d32-29889750-lvbrz"] Oct 30 18:30:26.753899 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.753877 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" Oct 30 18:30:26.810353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.810320 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f"] Oct 30 18:30:26.811408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811296 4883 cpu_manager.go:414] "containerMap: removing container" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" containerName="cleanup" Oct 30 18:30:26.811408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811318 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" containerName="cleanup" Oct 30 18:30:26.811408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811338 4883 cpu_manager.go:414] "containerMap: removing container" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" containerName="cleanup" Oct 30 18:30:26.811408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811346 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" containerName="cleanup" Oct 30 18:30:26.811408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811360 4883 cpu_manager.go:414] "containerMap: removing container" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" containerName="cleanup" Oct 30 18:30:26.811408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811368 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" containerName="cleanup" Oct 30 18:30:26.811408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811382 4883 cpu_manager.go:414] "containerMap: removing container" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" containerName="cleanup" Oct 30 18:30:26.811408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811390 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" containerName="cleanup" Oct 30 18:30:26.811408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811407 4883 cpu_manager.go:414] "containerMap: removing container" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" containerName="cleanup" Oct 30 18:30:26.811408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811414 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811431 4883 cpu_manager.go:414] "containerMap: removing container" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811438 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811449 4883 cpu_manager.go:414] "containerMap: removing container" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811455 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811473 4883 cpu_manager.go:414] "containerMap: removing container" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811481 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811666 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811687 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811697 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811708 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811718 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811728 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811736 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811745 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" containerName="cleanup" Oct 30 18:30:26.812036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.811754 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" containerName="cleanup" Oct 30 18:30:26.821652 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.821624 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f"] Oct 30 18:30:26.821799 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.821744 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:26.910171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.908592 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1041e24a-32c1-49ff-9c78-a1708404d7fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1041e24a-32c1-49ff-9c78-a1708404d7fd-maas-api-service-ca\") pod \"1041e24a-32c1-49ff-9c78-a1708404d7fd\" (UID: \"1041e24a-32c1-49ff-9c78-a1708404d7fd\") " Oct 30 18:30:26.910171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.908842 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77be19c9-c897-46ac-bcfc-da3e3623e2f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77be19c9-c897-46ac-bcfc-da3e3623e2f5-maas-api-service-ca\") pod \"77be19c9-c897-46ac-bcfc-da3e3623e2f5\" (UID: \"77be19c9-c897-46ac-bcfc-da3e3623e2f5\") " Oct 30 18:30:26.910171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.908909 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1041e24a-32c1-49ff-9c78-a1708404d7fd-kube-api-access-x294j\" (UniqueName: \"kubernetes.io/projected/1041e24a-32c1-49ff-9c78-a1708404d7fd-kube-api-access-x294j\") pod \"1041e24a-32c1-49ff-9c78-a1708404d7fd\" (UID: \"1041e24a-32c1-49ff-9c78-a1708404d7fd\") " Oct 30 18:30:26.910171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.909066 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77be19c9-c897-46ac-bcfc-da3e3623e2f5-kube-api-access-fg7c8\" (UniqueName: \"kubernetes.io/projected/77be19c9-c897-46ac-bcfc-da3e3623e2f5-kube-api-access-fg7c8\") pod \"77be19c9-c897-46ac-bcfc-da3e3623e2f5\" (UID: \"77be19c9-c897-46ac-bcfc-da3e3623e2f5\") " Oct 30 18:30:26.910171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.910120 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77be19c9-c897-46ac-bcfc-da3e3623e2f5-maas-api-service-ca" pod "77be19c9-c897-46ac-bcfc-da3e3623e2f5" (UID: "77be19c9-c897-46ac-bcfc-da3e3623e2f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:26.910798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.910570 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1041e24a-32c1-49ff-9c78-a1708404d7fd-maas-api-service-ca" pod "1041e24a-32c1-49ff-9c78-a1708404d7fd" (UID: "1041e24a-32c1-49ff-9c78-a1708404d7fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:26.913384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.913342 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77be19c9-c897-46ac-bcfc-da3e3623e2f5-kube-api-access-fg7c8" pod "77be19c9-c897-46ac-bcfc-da3e3623e2f5" (UID: "77be19c9-c897-46ac-bcfc-da3e3623e2f5"). InnerVolumeSpecName "kube-api-access-fg7c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:26.914513 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:26.914412 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1041e24a-32c1-49ff-9c78-a1708404d7fd-kube-api-access-x294j" pod "1041e24a-32c1-49ff-9c78-a1708404d7fd" (UID: "1041e24a-32c1-49ff-9c78-a1708404d7fd"). InnerVolumeSpecName "kube-api-access-x294j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.016178 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.016272 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.016297 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.016347 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.016367 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qq52\" (UniqueName: \"kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-kube-api-access-5qq52\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.016833 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.017057 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.017092 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.017186 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bacf95-f115-438a-a3cf-e7c753ec1a82-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.018364 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg7c8\" (UniqueName: \"kubernetes.io/projected/77be19c9-c897-46ac-bcfc-da3e3623e2f5-kube-api-access-fg7c8\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.018409 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1041e24a-32c1-49ff-9c78-a1708404d7fd-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.018430 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77be19c9-c897-46ac-bcfc-da3e3623e2f5-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:27.020030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.018445 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x294j\" (UniqueName: \"kubernetes.io/projected/1041e24a-32c1-49ff-9c78-a1708404d7fd-kube-api-access-x294j\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:27.124948 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.123326 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.142074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.123764 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.142645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.142606 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.143262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.143217 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.143975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.143858 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-data\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.144195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.144176 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bacf95-f115-438a-a3cf-e7c753ec1a82-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.145076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.144971 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bacf95-f115-438a-a3cf-e7c753ec1a82-istiod-ca-cert\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.145673 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.145233 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.145909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.145891 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.151018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.146354 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.151272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.146678 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-credential-socket\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.151489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.151471 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.151990 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.151888 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-certs\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.152147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.152128 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq52\" (UniqueName: \"kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-kube-api-access-5qq52\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.192700 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.170270 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-envoy\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.214347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.174721 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-podinfo\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.214487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.211110 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qq52\" (UniqueName: \"kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-kube-api-access-5qq52\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.214585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.211143 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" path="/var/lib/kubelet/pods/0213bbf6-2490-46de-a522-f12d2f4f2515/volumes" Oct 30 18:30:27.227456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.227404 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-token\") pod \"e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.427902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.427866 4883 generic.go:356] "Generic (PLEG): container finished" podID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" containerID="c6c42f3249c581e67fe1e62da71ac62da9246cf79ddd048f3d36d981d7ab31be" exitCode=60 Oct 30 18:30:27.428116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.427903 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" event={"ID":"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8","Type":"ContainerDied","Data":"c6c42f3249c581e67fe1e62da71ac62da9246cf79ddd048f3d36d981d7ab31be"} Oct 30 18:30:27.428116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.428027 4883 scope.go:122] "RemoveContainer" containerID="33c01c1a2ec2ccc9da0abb493051fe980a55d77b8391365fe7d82812292226cd" Oct 30 18:30:27.428480 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.428459 4883 scope.go:122] "RemoveContainer" containerID="c6c42f3249c581e67fe1e62da71ac62da9246cf79ddd048f3d36d981d7ab31be" Oct 30 18:30:27.429115 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:27.429084 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7_odh-ai-gateway-infra(8ac12c7a-1620-439a-93f1-eb8ac4a04aa8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" Oct 30 18:30:27.447568 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.447265 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:27.469211 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.469112 4883 generic.go:356] "Generic (PLEG): container finished" podID="96a19637-c44e-414a-b3b2-d963cf2ec937" containerID="d79504d2a9478ce60d70c509ba280dcb487c4197c3025dc9684601ed343e5254" exitCode=60 Oct 30 18:30:27.469366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.469232 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" event={"ID":"96a19637-c44e-414a-b3b2-d963cf2ec937","Type":"ContainerDied","Data":"d79504d2a9478ce60d70c509ba280dcb487c4197c3025dc9684601ed343e5254"} Oct 30 18:30:27.469808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.469786 4883 scope.go:122] "RemoveContainer" containerID="d79504d2a9478ce60d70c509ba280dcb487c4197c3025dc9684601ed343e5254" Oct 30 18:30:27.470235 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:27.470209 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8_odh-ai-gateway-infra(96a19637-c44e-414a-b3b2-d963cf2ec937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" Oct 30 18:30:27.487020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.486989 4883 generic.go:356] "Generic (PLEG): container finished" podID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" containerID="38756e377bb8d58d9f37f4bc3ad8b9a91bf3f5755a36c2a0ffb35e14b4dbae9e" exitCode=60 Oct 30 18:30:27.487194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.487107 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" event={"ID":"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607","Type":"ContainerDied","Data":"38756e377bb8d58d9f37f4bc3ad8b9a91bf3f5755a36c2a0ffb35e14b4dbae9e"} Oct 30 18:30:27.487543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.487523 4883 scope.go:122] "RemoveContainer" containerID="38756e377bb8d58d9f37f4bc3ad8b9a91bf3f5755a36c2a0ffb35e14b4dbae9e" Oct 30 18:30:27.488577 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:27.487863 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj_odh-ai-gateway-infra(2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" Oct 30 18:30:27.493415 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.493215 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" Oct 30 18:30:27.493415 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.493347 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h" event={"ID":"77be19c9-c897-46ac-bcfc-da3e3623e2f5","Type":"ContainerDied","Data":"502e7f35dddae5e3cebd90725ddd4e45901f216a48ed50f554e52691ae7a8100"} Oct 30 18:30:27.515259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.515164 4883 generic.go:356] "Generic (PLEG): container finished" podID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" containerID="7d5353cfb918034743563dff8516050e260b3d85a9f0bc8fb36348f9fcadb7b6" exitCode=60 Oct 30 18:30:27.515440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.515272 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" event={"ID":"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e","Type":"ContainerDied","Data":"7d5353cfb918034743563dff8516050e260b3d85a9f0bc8fb36348f9fcadb7b6"} Oct 30 18:30:27.517984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.515756 4883 scope.go:122] "RemoveContainer" containerID="7d5353cfb918034743563dff8516050e260b3d85a9f0bc8fb36348f9fcadb7b6" Oct 30 18:30:27.517984 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:27.516217 4883 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj_odh-ai-gateway-infra(add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" Oct 30 18:30:27.519018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.518994 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" event={"ID":"1041e24a-32c1-49ff-9c78-a1708404d7fd","Type":"ContainerDied","Data":"e1a9d13f21fc19d1ea3ac9b3235df89f1b4b36db66699e3ab8f1ed4c8f4db385"} Oct 30 18:30:27.519122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.519075 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx" Oct 30 18:30:27.521127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.521104 4883 scope.go:122] "RemoveContainer" containerID="138b1ee68d20f8a290bd1751c4b4b52ab415c92ba0f66ee49ef1b9bae857cca6" Oct 30 18:30:27.615831 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.606775 4883 scope.go:122] "RemoveContainer" containerID="800c6876bfc732821d26ee70c36ee408852c40e08692cacd8a5491d3946479d1" Oct 30 18:30:27.660531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.659985 4883 scope.go:122] "RemoveContainer" containerID="cc3e853ca9640c10056a90e9d53bdfc8be8bfae0af7c3371b57e93fec3e9ab2b" Oct 30 18:30:27.665335 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.664966 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h"] Oct 30 18:30:27.672967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.671972 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a2a10a-29889750-dpq5h"] Oct 30 18:30:27.672967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.672758 4883 scope.go:122] "RemoveContainer" containerID="739fb233a79e82c3ed324c00bae6e3bea2b1a5b36d70e3f4d7a258658c281d54" Oct 30 18:30:27.683407 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.683378 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8"] Oct 30 18:30:27.698296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.697959 4883 scope.go:122] "RemoveContainer" containerID="06ec51b19897c9e588cc57aff6811c7f55b84b50ecb4b7c9782cbb17f7dd9396" Oct 30 18:30:27.708959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.708344 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj"] Oct 30 18:30:27.719735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.719674 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx"] Oct 30 18:30:27.729526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.728191 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:27.729792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.729695 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6e8fca-29889750-vccbx"] Oct 30 18:30:27.729792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.729740 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:27.731368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.731326 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:30:27.803194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.803039 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" Oct 30 18:30:27.817622 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:30:27.817594 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91bacf95_f115_438a_a3cf_e7c753ec1a82.slice/crio-dbb6ebd27ff13905a9209299cf00703fda1d16c253fd50a4656d581ba9cfd337 WatchSource:0}: Error finding container dbb6ebd27ff13905a9209299cf00703fda1d16c253fd50a4656d581ba9cfd337: Status 404 returned error can't find the container with id dbb6ebd27ff13905a9209299cf00703fda1d16c253fd50a4656d581ba9cfd337 Oct 30 18:30:27.817622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.817600 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f"] Oct 30 18:30:27.820429 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.820399 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:30:27.820519 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.820459 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:30:27.820519 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.820487 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:30:27.924450 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.924415 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/888e9299-97b3-4cef-9213-69663ac795b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888e9299-97b3-4cef-9213-69663ac795b0-maas-api-service-ca\") pod \"888e9299-97b3-4cef-9213-69663ac795b0\" (UID: \"888e9299-97b3-4cef-9213-69663ac795b0\") " Oct 30 18:30:27.924577 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.924492 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888e9299-97b3-4cef-9213-69663ac795b0-kube-api-access-ctv8s\" (UniqueName: \"kubernetes.io/projected/888e9299-97b3-4cef-9213-69663ac795b0-kube-api-access-ctv8s\") pod \"888e9299-97b3-4cef-9213-69663ac795b0\" (UID: \"888e9299-97b3-4cef-9213-69663ac795b0\") " Oct 30 18:30:27.926195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.926164 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888e9299-97b3-4cef-9213-69663ac795b0-maas-api-service-ca" pod "888e9299-97b3-4cef-9213-69663ac795b0" (UID: "888e9299-97b3-4cef-9213-69663ac795b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:27.927989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:27.927917 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888e9299-97b3-4cef-9213-69663ac795b0-kube-api-access-ctv8s" pod "888e9299-97b3-4cef-9213-69663ac795b0" (UID: "888e9299-97b3-4cef-9213-69663ac795b0"). InnerVolumeSpecName "kube-api-access-ctv8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:28.026008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:28.025976 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888e9299-97b3-4cef-9213-69663ac795b0-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:28.026008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:28.026007 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctv8s\" (UniqueName: \"kubernetes.io/projected/888e9299-97b3-4cef-9213-69663ac795b0-kube-api-access-ctv8s\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:28.498240 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:28.498208 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7"] Oct 30 18:30:28.530718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:28.530664 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" event={"ID":"91bacf95-f115-438a-a3cf-e7c753ec1a82","Type":"ContainerStarted","Data":"424530e746905bfbcfe817a84327f9373478767c3135b18159b547e0aafd903b"} Oct 30 18:30:28.530718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:28.530722 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" event={"ID":"91bacf95-f115-438a-a3cf-e7c753ec1a82","Type":"ContainerStarted","Data":"dbb6ebd27ff13905a9209299cf00703fda1d16c253fd50a4656d581ba9cfd337"} Oct 30 18:30:28.536537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:28.536513 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" Oct 30 18:30:28.536638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:28.536518 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj" event={"ID":"888e9299-97b3-4cef-9213-69663ac795b0","Type":"ContainerDied","Data":"19396b49c7eb8c5ddd57446ac3a214c89c1f251939e4a1607ec341ebb56951e4"} Oct 30 18:30:28.536699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:28.536652 4883 scope.go:122] "RemoveContainer" containerID="800fbffbaf5055155e5e6095df9a70a837d2fd886ff02e555ac356b8eba55ee1" Oct 30 18:30:28.603867 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:28.601822 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" podStartSLOduration=2.601798414 podStartE2EDuration="2.601798414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:28.588350036 +0000 UTC m=+2287.545353873" watchObservedRunningTime="2026-10-30 18:30:28.601798414 +0000 UTC m=+2287.558802251" Oct 30 18:30:28.782072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:28.781998 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj"] Oct 30 18:30:28.822263 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:28.816703 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc458-29889750-8nmkj"] Oct 30 18:30:28.829685 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:28.828362 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj"] Oct 30 18:30:29.221047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.208244 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" Oct 30 18:30:29.310971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.307763 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-kube-api-access-cp2mz\" (UniqueName: \"kubernetes.io/projected/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-kube-api-access-cp2mz\") pod \"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8\" (UID: \"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8\") " Oct 30 18:30:29.310971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.307896 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-maas-api-service-ca\") pod \"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8\" (UID: \"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8\") " Oct 30 18:30:29.357505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.352707 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-maas-api-service-ca" pod "8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" (UID: "8ac12c7a-1620-439a-93f1-eb8ac4a04aa8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:29.410180 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.409742 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:29.550536 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.550491 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-kube-api-access-cp2mz" pod "8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" (UID: "8ac12c7a-1620-439a-93f1-eb8ac4a04aa8"). InnerVolumeSpecName "kube-api-access-cp2mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:29.553002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.552973 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 18:30:29.574867 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.574573 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" podUID="91bacf95-f115-438a-a3cf-e7c753ec1a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 18:30:29.617840 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.617787 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1041e24a-32c1-49ff-9c78-a1708404d7fd" path="/var/lib/kubelet/pods/1041e24a-32c1-49ff-9c78-a1708404d7fd/volumes" Oct 30 18:30:29.669964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.664377 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77be19c9-c897-46ac-bcfc-da3e3623e2f5" path="/var/lib/kubelet/pods/77be19c9-c897-46ac-bcfc-da3e3623e2f5/volumes" Oct 30 18:30:29.670588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.670069 4883 scope.go:122] "RemoveContainer" containerID="b4529a30d4e0f37c55f731d2e0b7815f3fee16e21f5050a1ada09c02b08d355e" Oct 30 18:30:29.686116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.682156 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp2mz\" (UniqueName: \"kubernetes.io/projected/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8-kube-api-access-cp2mz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:29.686116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.682878 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="888e9299-97b3-4cef-9213-69663ac795b0" path="/var/lib/kubelet/pods/888e9299-97b3-4cef-9213-69663ac795b0/volumes" Oct 30 18:30:29.686116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.684876 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:29.742508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.691268 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" event={"ID":"8ac12c7a-1620-439a-93f1-eb8ac4a04aa8","Type":"ContainerDied","Data":"7dda001f53bebb14337cafa48c03a9d4122cf0bab98f40696913c79fc93f46b2"} Oct 30 18:30:29.742508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.691356 4883 scope.go:122] "RemoveContainer" containerID="c6c42f3249c581e67fe1e62da71ac62da9246cf79ddd048f3d36d981d7ab31be" Oct 30 18:30:29.742508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.691855 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7" Oct 30 18:30:29.742508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.719179 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" Oct 30 18:30:29.800678 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.800626 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a19637-c44e-414a-b3b2-d963cf2ec937-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a19637-c44e-414a-b3b2-d963cf2ec937-maas-api-service-ca\") pod \"96a19637-c44e-414a-b3b2-d963cf2ec937\" (UID: \"96a19637-c44e-414a-b3b2-d963cf2ec937\") " Oct 30 18:30:29.827654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.827622 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a19637-c44e-414a-b3b2-d963cf2ec937-maas-api-service-ca" pod "96a19637-c44e-414a-b3b2-d963cf2ec937" (UID: "96a19637-c44e-414a-b3b2-d963cf2ec937"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:29.827885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.827867 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a19637-c44e-414a-b3b2-d963cf2ec937-kube-api-access-tpfbz\" (UniqueName: \"kubernetes.io/projected/96a19637-c44e-414a-b3b2-d963cf2ec937-kube-api-access-tpfbz\") pod \"96a19637-c44e-414a-b3b2-d963cf2ec937\" (UID: \"96a19637-c44e-414a-b3b2-d963cf2ec937\") " Oct 30 18:30:29.829334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.829312 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a19637-c44e-414a-b3b2-d963cf2ec937-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:29.831753 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.830729 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" Oct 30 18:30:29.836050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.835857 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a19637-c44e-414a-b3b2-d963cf2ec937-kube-api-access-tpfbz" pod "96a19637-c44e-414a-b3b2-d963cf2ec937" (UID: "96a19637-c44e-414a-b3b2-d963cf2ec937"). InnerVolumeSpecName "kube-api-access-tpfbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:29.845528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.845501 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" event={"ID":"96a19637-c44e-414a-b3b2-d963cf2ec937","Type":"ContainerDied","Data":"c65b9c6ee1534e908ccf4a981ad8915422872f040b95ce055b6ecc908d196a8b"} Oct 30 18:30:29.845784 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.845768 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8" Oct 30 18:30:29.857246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.857220 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" Oct 30 18:30:29.857388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.857365 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj" event={"ID":"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607","Type":"ContainerDied","Data":"d5aa5f5de8d920b558028f3949b6a84ddd65e6a5ed83d2a3e81b310b42c1cd2b"} Oct 30 18:30:29.863693 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.863669 4883 scope.go:122] "RemoveContainer" containerID="d79504d2a9478ce60d70c509ba280dcb487c4197c3025dc9684601ed343e5254" Oct 30 18:30:29.930058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.930023 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-maas-api-service-ca\") pod \"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607\" (UID: \"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607\") " Oct 30 18:30:29.930187 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.930082 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-kube-api-access-69lqp\" (UniqueName: \"kubernetes.io/projected/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-kube-api-access-69lqp\") pod \"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607\" (UID: \"2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607\") " Oct 30 18:30:29.930896 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.930870 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpfbz\" (UniqueName: \"kubernetes.io/projected/96a19637-c44e-414a-b3b2-d963cf2ec937-kube-api-access-tpfbz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:29.931615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.931581 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-maas-api-service-ca" pod "2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" (UID: "2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:29.934543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.934300 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-kube-api-access-69lqp" pod "2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" (UID: "2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607"). InnerVolumeSpecName "kube-api-access-69lqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:29.943051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.943012 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" Oct 30 18:30:29.950370 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:29.949319 4883 scope.go:122] "RemoveContainer" containerID="38756e377bb8d58d9f37f4bc3ad8b9a91bf3f5755a36c2a0ffb35e14b4dbae9e" Oct 30 18:30:30.025761 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.025383 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7"] Oct 30 18:30:30.032715 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.032683 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-602e18-29889750-8w6l7"] Oct 30 18:30:30.033205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.033184 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-maas-api-service-ca\") pod \"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e\" (UID: \"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e\") " Oct 30 18:30:30.033340 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.033229 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-kube-api-access-7zzbm\" (UniqueName: \"kubernetes.io/projected/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-kube-api-access-7zzbm\") pod \"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e\" (UID: \"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e\") " Oct 30 18:30:30.033753 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.033730 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-maas-api-service-ca" pod "add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" (UID: "add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:30.034149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.034126 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:30.034265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.034251 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-maas-api-service-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:30.034360 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.034347 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69lqp\" (UniqueName: \"kubernetes.io/projected/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607-kube-api-access-69lqp\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:30.036370 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.036342 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-kube-api-access-7zzbm" pod "add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" (UID: "add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e"). InnerVolumeSpecName "kube-api-access-7zzbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:30.101983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.101910 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8"] Oct 30 18:30:30.108445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.108413 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ac632e-29889750-zbmd8"] Oct 30 18:30:30.136441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.136410 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zzbm\" (UniqueName: \"kubernetes.io/projected/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e-kube-api-access-7zzbm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:30:30.254985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.254911 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj"] Oct 30 18:30:30.262477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.262442 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b424a5-29889750-5rfhj"] Oct 30 18:30:30.459961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.452315 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 18:30:30.459961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.452393 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" podUID="91bacf95-f115-438a-a3cf-e7c753ec1a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 18:30:30.906195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.906133 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" event={"ID":"add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e","Type":"ContainerDied","Data":"94ad5d6556381247760043da1f9c53be3c76173f20f67f0d6b11cddd913f335b"} Oct 30 18:30:30.906733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.906239 4883 scope.go:122] "RemoveContainer" containerID="7d5353cfb918034743563dff8516050e260b3d85a9f0bc8fb36348f9fcadb7b6" Oct 30 18:30:30.906733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.906353 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj" Oct 30 18:30:30.989998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:30.988385 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj"] Oct 30 18:30:31.007334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:31.007305 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-50e9fc-29889750-bx4fj"] Oct 30 18:30:31.198334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:31.198245 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" path="/var/lib/kubelet/pods/2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607/volumes" Oct 30 18:30:31.198880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:31.198857 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" path="/var/lib/kubelet/pods/8ac12c7a-1620-439a-93f1-eb8ac4a04aa8/volumes" Oct 30 18:30:31.199410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:31.199388 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" path="/var/lib/kubelet/pods/96a19637-c44e-414a-b3b2-d963cf2ec937/volumes" Oct 30 18:30:31.200159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:31.200140 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" path="/var/lib/kubelet/pods/add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e/volumes" Oct 30 18:30:31.448712 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:31.448622 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 18:30:31.449001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:31.448972 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" podUID="91bacf95-f115-438a-a3cf-e7c753ec1a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 18:30:32.448550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:32.448511 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 18:30:32.448987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:32.448567 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" podUID="91bacf95-f115-438a-a3cf-e7c753ec1a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 18:30:33.448706 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:33.448623 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 18:30:33.448706 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:33.448681 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" podUID="91bacf95-f115-438a-a3cf-e7c753ec1a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 18:30:34.448382 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:34.448343 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 18:30:34.448605 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:34.448411 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" podUID="91bacf95-f115-438a-a3cf-e7c753ec1a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 18:30:35.083027 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:30:35.082983 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:30:35.459013 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:35.458838 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:35.459319 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:35.459284 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:35.460711 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:35.460678 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:30:36.245443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.245408 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg"] Oct 30 18:30:36.246359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246337 4883 cpu_manager.go:414] "containerMap: removing container" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" containerName="cleanup" Oct 30 18:30:36.246432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246363 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" containerName="cleanup" Oct 30 18:30:36.246432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246379 4883 cpu_manager.go:414] "containerMap: removing container" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" containerName="cleanup" Oct 30 18:30:36.246432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246388 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" containerName="cleanup" Oct 30 18:30:36.246432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246406 4883 cpu_manager.go:414] "containerMap: removing container" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" containerName="cleanup" Oct 30 18:30:36.246432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246414 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246434 4883 cpu_manager.go:414] "containerMap: removing container" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246441 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246462 4883 cpu_manager.go:414] "containerMap: removing container" podUID="888e9299-97b3-4cef-9213-69663ac795b0" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246470 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888e9299-97b3-4cef-9213-69663ac795b0" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246482 4883 cpu_manager.go:414] "containerMap: removing container" podUID="888e9299-97b3-4cef-9213-69663ac795b0" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246489 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888e9299-97b3-4cef-9213-69663ac795b0" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246502 4883 cpu_manager.go:414] "containerMap: removing container" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246509 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246520 4883 cpu_manager.go:414] "containerMap: removing container" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246527 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246554 4883 cpu_manager.go:414] "containerMap: removing container" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246561 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246580 4883 cpu_manager.go:414] "containerMap: removing container" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246587 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0213bbf6-2490-46de-a522-f12d2f4f2515" containerName="cleanup" Oct 30 18:30:36.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246597 4883 cpu_manager.go:414] "containerMap: removing container" podUID="888e9299-97b3-4cef-9213-69663ac795b0" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246604 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888e9299-97b3-4cef-9213-69663ac795b0" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246618 4883 cpu_manager.go:414] "containerMap: removing container" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246625 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246643 4883 cpu_manager.go:414] "containerMap: removing container" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246650 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246795 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246810 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246821 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246831 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246840 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="888e9299-97b3-4cef-9213-69663ac795b0" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246848 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="888e9299-97b3-4cef-9213-69663ac795b0" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246857 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246864 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246877 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="888e9299-97b3-4cef-9213-69663ac795b0" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246888 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" containerName="cleanup" Oct 30 18:30:36.247115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.246900 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" containerName="cleanup" Oct 30 18:30:36.252854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.252827 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.272828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.272794 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg"] Oct 30 18:30:36.279439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.279410 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ac632e\"/\"kube-root-ca.crt\"" Oct 30 18:30:36.279602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.279410 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ac632e\"/\"openshift-service-ca.crt\"" Oct 30 18:30:36.286622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.286594 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ac632e\"/\"auth-test-model-ac632e-kserve-self-signed-certs\"" Oct 30 18:30:36.286761 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.286628 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ac632e\"/\"default-dockercfg-mbqlf\"" Oct 30 18:30:36.330748 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.330703 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-dshm\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.330993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.330964 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tls-certs\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.331131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.331104 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tmp-dir\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.331589 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.331232 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-home\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.331589 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.331327 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-model-cache\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.331589 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.331380 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnwkq\" (UniqueName: \"kubernetes.io/projected/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-kube-api-access-bnwkq\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.432672 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.432633 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-dshm\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.432879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.432724 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tls-certs\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.432879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.432786 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tmp-dir\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.432879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.432850 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-home\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.433099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.432898 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-model-cache\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.433099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.432959 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnwkq\" (UniqueName: \"kubernetes.io/projected/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-kube-api-access-bnwkq\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.433218 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.433203 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tmp-dir\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.433276 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.433242 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-model-cache\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.433586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.433562 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-home\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.435114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.435092 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-dshm\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.436525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.436497 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tls-certs\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.505663 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.505578 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnwkq\" (UniqueName: \"kubernetes.io/projected/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-kube-api-access-bnwkq\") pod \"auth-test-model-ac632e-kserve-555fd95b85-kjqwg\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.587001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.585078 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:36.995846 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:30:36.995817 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae5d200e_7bb3_4a5e_9890_29eeeaa67ef7.slice/crio-9874896420e54aca35838aa76f34c9eea13b231dfaf96ef0fa6bd423ad432f67 WatchSource:0}: Error finding container 9874896420e54aca35838aa76f34c9eea13b231dfaf96ef0fa6bd423ad432f67: Status 404 returned error can't find the container with id 9874896420e54aca35838aa76f34c9eea13b231dfaf96ef0fa6bd423ad432f67 Oct 30 18:30:36.995980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:36.995926 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg"] Oct 30 18:30:37.989864 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:37.989824 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" event={"ID":"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7","Type":"ContainerStarted","Data":"b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17"} Oct 30 18:30:37.990351 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:37.989872 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" event={"ID":"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7","Type":"ContainerStarted","Data":"9874896420e54aca35838aa76f34c9eea13b231dfaf96ef0fa6bd423ad432f67"} Oct 30 18:30:38.036989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:38.036909 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podStartSLOduration=2.036894467 podStartE2EDuration="2.036894467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:38.033922356 +0000 UTC m=+2296.990926192" watchObservedRunningTime="2026-10-30 18:30:38.036894467 +0000 UTC m=+2296.993898281" Oct 30 18:30:38.586243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:38.586198 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:46.585586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:46.585550 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:46.598698 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:46.598667 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:47.096745 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:47.095663 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:30:53.036755 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:53.036724 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6"] Oct 30 18:30:53.050699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:53.050672 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2343eca5-78kv6"] Oct 30 18:30:53.179088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:30:53.179059 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2fb32e-690f-41d3-a07c-074cdb76f0d6" path="/var/lib/kubelet/pods/6c2fb32e-690f-41d3-a07c-074cdb76f0d6/volumes" Oct 30 18:31:02.046038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:02.046002 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms"] Oct 30 18:31:02.058984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:02.058951 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63c852d1-hbfms"] Oct 30 18:31:03.179665 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:03.179632 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc985ab4-6938-44a5-9a9c-89530885929e" path="/var/lib/kubelet/pods/bc985ab4-6938-44a5-9a9c-89530885929e/volumes" Oct 30 18:31:05.079725 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:31:05.079685 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:31:17.033740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:17.033704 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz"] Oct 30 18:31:17.046854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:17.046823 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb7d75f-pg2dz"] Oct 30 18:31:17.178897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:17.178860 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0" path="/var/lib/kubelet/pods/20aed91f-cb4d-4f9d-bb1a-fc8ca3b0aad0/volumes" Oct 30 18:31:19.037034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:19.036998 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp"] Oct 30 18:31:19.050207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:19.050174 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b410d4a0-7tsxp"] Oct 30 18:31:19.180084 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:19.180052 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d6d4e5-8996-4e7c-ad42-1eeff776df7f" path="/var/lib/kubelet/pods/64d6d4e5-8996-4e7c-ad42-1eeff776df7f/volumes" Oct 30 18:31:28.453635 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.453598 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d"] Oct 30 18:31:28.454241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.454211 4883 cpu_manager.go:414] "containerMap: removing container" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" containerName="cleanup" Oct 30 18:31:28.454241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.454228 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" containerName="cleanup" Oct 30 18:31:28.454366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.454258 4883 cpu_manager.go:414] "containerMap: removing container" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" containerName="cleanup" Oct 30 18:31:28.454366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.454264 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" containerName="cleanup" Oct 30 18:31:28.454366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.454282 4883 cpu_manager.go:414] "containerMap: removing container" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" containerName="cleanup" Oct 30 18:31:28.454366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.454289 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" containerName="cleanup" Oct 30 18:31:28.454590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.454405 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac12c7a-1620-439a-93f1-eb8ac4a04aa8" containerName="cleanup" Oct 30 18:31:28.454590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.454416 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="add1fe29-8f2a-49e5-9e2f-31a4e0d92c5e" containerName="cleanup" Oct 30 18:31:28.454590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.454423 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4751dd-4cdc-4e8e-9bd0-8bf7fd99b607" containerName="cleanup" Oct 30 18:31:28.454590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.454433 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a19637-c44e-414a-b3b2-d963cf2ec937" containerName="cleanup" Oct 30 18:31:28.459411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.459385 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.479966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.479910 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d"] Oct 30 18:31:28.488421 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.488380 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-5cb3d5\"/\"auth-test-model-5cb3d5-kserve-self-signed-certs\"" Oct 30 18:31:28.488574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.488459 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-5cb3d5\"/\"default-dockercfg-87vdn\"" Oct 30 18:31:28.488980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.488930 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-5cb3d5\"/\"openshift-service-ca.crt\"" Oct 30 18:31:28.488980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.488966 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-5cb3d5\"/\"kube-root-ca.crt\"" Oct 30 18:31:28.608759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.608723 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-dshm\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.608989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.608774 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tls-certs\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.608989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.608878 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-home\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.608989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.608955 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-988b7\" (UniqueName: \"kubernetes.io/projected/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-kube-api-access-988b7\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.609194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.609017 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tmp-dir\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.609194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.609174 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-model-cache\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.710603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.710504 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tls-certs\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.711853 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.711826 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-home\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.712055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.712037 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-988b7\" (UniqueName: \"kubernetes.io/projected/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-kube-api-access-988b7\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.712195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.712178 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tmp-dir\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.712421 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.712404 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-model-cache\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.712635 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.712620 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-dshm\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.713789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.713726 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-home\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.714737 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.714714 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tmp-dir\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.715117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.715097 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-model-cache\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.742311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.742276 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tls-certs\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.749021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.748985 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-dshm\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.760044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.752568 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-988b7\" (UniqueName: \"kubernetes.io/projected/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-kube-api-access-988b7\") pod \"auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:28.777721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:28.777336 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:29.273842 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:31:29.273795 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a7f3b3e_d9a9_4c88_a79e_06fba5eb9d96.slice/crio-d58be5277a448f38651c41c4ca5ab6d5aa46445ffc12adc4f4e18e744f47ec48 WatchSource:0}: Error finding container d58be5277a448f38651c41c4ca5ab6d5aa46445ffc12adc4f4e18e744f47ec48: Status 404 returned error can't find the container with id d58be5277a448f38651c41c4ca5ab6d5aa46445ffc12adc4f4e18e744f47ec48 Oct 30 18:31:29.277003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:29.276974 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d"] Oct 30 18:31:29.482643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:29.482605 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" event={"ID":"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96","Type":"ContainerStarted","Data":"1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff"} Oct 30 18:31:29.483069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:29.482653 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" event={"ID":"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96","Type":"ContainerStarted","Data":"d58be5277a448f38651c41c4ca5ab6d5aa46445ffc12adc4f4e18e744f47ec48"} Oct 30 18:31:29.533871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:29.533752 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podStartSLOduration=1.533737217 podStartE2EDuration="1.533737217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:29.529820562 +0000 UTC m=+2348.486824396" watchObservedRunningTime="2026-10-30 18:31:29.533737217 +0000 UTC m=+2348.490741424" Oct 30 18:31:29.778213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:29.778167 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:29.985969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:29.985922 4883 scope.go:122] "RemoveContainer" containerID="d10b6721b3bebde09fdc9ca1ee42e80815657a889a3e29b82e33d2f02ffa1aac" Oct 30 18:31:29.999749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:29.999722 4883 scope.go:122] "RemoveContainer" containerID="45224494045f6e6e541ecb6fded352b87c2d1218f50208a5d67ff5f0c6f90b35" Oct 30 18:31:30.009801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:30.009777 4883 scope.go:122] "RemoveContainer" containerID="6591233accedeef2c7961d264812c151d16da86e94a7195e67494bf06f2c051a" Oct 30 18:31:30.019134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:30.019111 4883 scope.go:122] "RemoveContainer" containerID="66310b7d1a6fb816e4f1a083a21d20f8a26bab7f0414586fc1efd657eb3efa3a" Oct 30 18:31:35.079558 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:31:35.079519 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:31:38.778923 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:38.778890 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:38.793145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:38.793121 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:39.772001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:39.771257 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:41.076960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:41.076249 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh"] Oct 30 18:31:41.115960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:41.112116 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77f47639-xdmqh"] Oct 30 18:31:41.272959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:41.260602 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3184250-2320-4ab5-a678-d96b88c9333d" path="/var/lib/kubelet/pods/e3184250-2320-4ab5-a678-d96b88c9333d/volumes" Oct 30 18:31:41.411968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:41.410493 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:41.501958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:41.501041 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:41.532958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:41.532074 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:41.563960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:41.563102 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:41.613949 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:41.613873 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:41.879958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:41.876036 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.063968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.063183 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t"] Oct 30 18:31:42.080147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.080084 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.105957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.104314 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-332a9e40-9mx4t"] Oct 30 18:31:42.131099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.131002 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:31:42.270083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.270018 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.289072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.289025 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.431134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.431024 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:31:42.432167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.432138 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:31:42.501961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.501342 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:31:42.502177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.501964 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:31:42.520558 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.519950 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.557959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.552798 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.557959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.553607 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:31:42.585696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.585649 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.586522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.586496 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:31:42.630137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.627529 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": context deadline exceeded" Oct 30 18:31:42.630137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.628202 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:42.711187 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.710465 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:31:42.829690 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.829558 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:31:42.877413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.877368 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:42.878919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:42.878894 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:31:43.011356 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.008630 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:43.117960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.093444 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:43.145037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.119223 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:31:43.162338 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.162291 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:31:43.253038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.172801 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:31:43.345585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.342130 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:43.348523 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.347656 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:31:43.412968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.409246 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf54ed9d-3143-422f-82ac-349c2e32b3e1" path="/var/lib/kubelet/pods/cf54ed9d-3143-422f-82ac-349c2e32b3e1/volumes" Oct 30 18:31:43.412968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.409751 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:31:43.412968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.409817 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:31:43.412968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.409857 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:31:43.512986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.512028 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:31:43.568242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.539309 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:43.593265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.593235 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:31:43.609666 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.591068 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:31:43.637957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.632022 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:43.733960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.724200 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:43.733960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.725050 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:31:43.848404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.848364 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:43.849161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.849134 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:31:43.880320 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:43.880228 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.185048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:44.144301 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.198869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:44.185620 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:31:44.241961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:44.200991 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:31:44.264072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:44.264024 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.624133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:44.343190 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.638086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:44.378655 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:31:44.645973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:44.572836 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:31:44.645973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:44.615003 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:31:44.645973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:44.620190 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.725330 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:44.725284 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:44.727703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:44.727651 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:31:44.853665 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:44.853563 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:31:45.015728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.015139 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:31:45.310150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.310041 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.481148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.364635 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.483691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.364814 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 18:31:45.643445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.643389 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:31:45.723961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.712053 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.723961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.712462 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:31:45.723961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.717338 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:31:45.723961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.717735 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.723961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.718027 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:31:45.723961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.718680 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:31:45.723961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.718960 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.724671 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.724640 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.725697 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.725667 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:45.859147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:45.859092 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.017207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.016158 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.486845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.485052 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.486845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.485441 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:31:46.491152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.491033 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:31:46.491152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.491094 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.535421 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.535366 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.536093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.536068 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:31:46.649914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.649865 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:31:46.723165 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.723106 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": context deadline exceeded" Oct 30 18:31:46.724251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.724227 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:46.751034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.725784 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 30 18:31:46.773973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.769138 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:31:46.773973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.753058 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:31:46.773973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.769809 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:31:46.796971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.753021 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.796971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.753122 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:31:46.796971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.753712 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:31:46.796971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.753078 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:31:46.796971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.753103 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:46.974103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:46.961182 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded" Oct 30 18:31:47.135381 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.110919 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:31:47.135381 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.112851 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:31:47.504019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.486580 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.504019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.498367 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 18:31:47.537600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.537550 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.608984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.608948 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:31:47.670385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.670341 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.702124 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.702074 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.725106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.725058 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.809673 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.809576 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:31:47.832877 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.830312 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:31:47.862962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.840370 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:47.862962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.841405 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:31:47.862962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.858866 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:31:47.867365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:47.867322 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.043740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.043706 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:31:48.165628 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.148902 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:31:48.313036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.177166 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:31:48.314571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.235724 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.454362 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.454258 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.493575 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.493529 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.509957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.506171 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.543401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.543354 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:31:48.567777 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.547426 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.711202 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.711099 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.729102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.729044 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": context deadline exceeded" Oct 30 18:31:48.766241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.766208 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:31:48.821794 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.820449 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.842112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.842052 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.842779 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.842734 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:31:48.872236 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.872194 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:48.872843 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:48.872480 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.077024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.070065 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.077024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.070316 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:31:49.258255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.258213 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:31:49.323584 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.323536 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.332014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.331837 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:31:49.344021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.343769 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.652750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.577596 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:31:49.652750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.587067 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 18:31:49.652750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.588225 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:31:49.652750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.589328 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:31:49.830058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.828594 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.830058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.829209 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.830058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.829468 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.841343 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.831286 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.843510 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.843454 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:31:49.844246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.844212 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.878297 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.876194 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:49.878297 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:49.877014 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:31:50.081179 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.081076 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.089967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.081633 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:31:50.331438 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.331333 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.332225 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.332188 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.532196 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.523125 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.575083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.575034 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 18:31:50.600019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.579214 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:31:50.600019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.591085 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.600019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.591485 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 18:31:50.659428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.659284 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.660051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.660003 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.875000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.863077 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:31:50.875000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.863430 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:31:50.875000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.866113 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.875000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.866379 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:31:50.882386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.882008 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:50.882386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:50.882205 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:31:51.070341 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.069247 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:31:51.084684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.083998 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx"] Oct 30 18:31:51.084684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.084107 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:51.146171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.141098 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9a51f9-678wx"] Oct 30 18:31:51.427022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.383023 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:31:51.427022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.383459 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:31:51.530265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.530213 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:31:51.579635 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.579592 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:51.602957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.597371 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02aa44b5-7d4b-46c3-811a-5cca736fe4cb" path="/var/lib/kubelet/pods/02aa44b5-7d4b-46c3-811a-5cca736fe4cb/volumes" Oct 30 18:31:51.626768 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.626734 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:31:51.649024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.609708 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:51.666328 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.664017 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:51.666328 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.664215 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:31:51.863340 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.863290 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:51.864091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.864051 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:51.883966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.871218 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:51.884221 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.884186 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:51.900309 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.885222 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:51.900309 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:51.885492 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:31:52.182016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.166569 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6"] Oct 30 18:31:52.182016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.166730 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:31:52.185744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.182496 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:31:52.197252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.197202 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:31:52.315843 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.315761 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e35b3c8-gvwq6"] Oct 30 18:31:52.327140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.327113 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:31:52.327796 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.327772 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:31:52.333035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.333015 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:31:52.333667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.333651 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:31:52.352992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.352967 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:31:52.353307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.353292 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:31:52.354989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.354929 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:31:52.355094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.355037 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:31:52.398457 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.398427 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:31:52.400016 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.399966 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:31:52.403285 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:52.403259 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:31:53.205120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:53.205085 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db09c429-cef5-4171-ad7e-8499fdf09288" path="/var/lib/kubelet/pods/db09c429-cef5-4171-ad7e-8499fdf09288/volumes" Oct 30 18:31:58.099600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:58.099568 4883 generic.go:356] "Generic (PLEG): container finished" podID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerID="40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181" exitCode=0 Oct 30 18:31:58.100062 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:58.099630 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" event={"ID":"63bb0b62-59e5-4aa2-b5d1-c8837731bf53","Type":"ContainerDied","Data":"40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181"} Oct 30 18:31:58.100111 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:58.100084 4883 scope.go:122] "RemoveContainer" containerID="40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181" Oct 30 18:31:58.101812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:58.101775 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:31:58.101920 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:58.101865 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:31:58.102008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:58.101929 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:31:59.109909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:31:59.109876 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" event={"ID":"63bb0b62-59e5-4aa2-b5d1-c8837731bf53","Type":"ContainerStarted","Data":"a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb"} Oct 30 18:32:00.102510 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:00.102467 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:32:00.103923 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:00.103881 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 18:32:00.104079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:00.103965 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 18:32:01.102760 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:01.102722 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 18:32:01.103190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:01.102787 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 18:32:01.916710 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:01.916677 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:32:02.104388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:02.104347 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 18:32:02.105110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:02.105073 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 18:32:03.102705 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:03.102626 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 18:32:03.102705 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:03.102687 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 18:32:04.031500 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:04.031466 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9"] Oct 30 18:32:04.044481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:04.044449 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e77bc3a-gbwr9"] Oct 30 18:32:04.102811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:04.102780 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Oct 30 18:32:04.103005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:04.102833 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Oct 30 18:32:05.078249 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:32:05.078214 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:32:05.107809 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:05.107781 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:32:05.109048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:05.109028 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:32:05.180766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:05.180733 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a40295a-67e7-4173-aca6-6354a8dd2fc6" path="/var/lib/kubelet/pods/3a40295a-67e7-4173-aca6-6354a8dd2fc6/volumes" Oct 30 18:32:07.631079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.631035 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq"] Oct 30 18:32:07.635268 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.635245 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:07.651724 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.651699 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:32:07.665151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.665096 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:32:07.665151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.665092 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:32:07.665456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.665437 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:32:07.672465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.672433 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq"] Oct 30 18:32:07.692552 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.692524 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq\" (UID: \"d1c3873b-0c90-4787-99a8-b8ba05807910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:07.692687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.692643 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq\" (UID: \"d1c3873b-0c90-4787-99a8-b8ba05807910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:07.692733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.692681 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds56n\" (UniqueName: \"kubernetes.io/projected/d1c3873b-0c90-4787-99a8-b8ba05807910-kube-api-access-ds56n\") pod \"maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq\" (UID: \"d1c3873b-0c90-4787-99a8-b8ba05807910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:07.794682 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.794645 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq\" (UID: \"d1c3873b-0c90-4787-99a8-b8ba05807910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:07.794857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.794738 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq\" (UID: \"d1c3873b-0c90-4787-99a8-b8ba05807910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:07.794857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.794767 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds56n\" (UniqueName: \"kubernetes.io/projected/d1c3873b-0c90-4787-99a8-b8ba05807910-kube-api-access-ds56n\") pod \"maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq\" (UID: \"d1c3873b-0c90-4787-99a8-b8ba05807910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:07.795323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.795301 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq\" (UID: \"d1c3873b-0c90-4787-99a8-b8ba05807910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:07.797213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.797190 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq\" (UID: \"d1c3873b-0c90-4787-99a8-b8ba05807910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:07.820568 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.820538 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds56n\" (UniqueName: \"kubernetes.io/projected/d1c3873b-0c90-4787-99a8-b8ba05807910-kube-api-access-ds56n\") pod \"maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq\" (UID: \"d1c3873b-0c90-4787-99a8-b8ba05807910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:07.948428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:07.948343 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:08.258718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:08.258685 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq"] Oct 30 18:32:08.260006 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:32:08.259975 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c3873b_0c90_4787_99a8_b8ba05807910.slice/crio-44a88b7002386ae057d45d14eb7956f96bbbe51e93dbc550e2114cf295b5ebdf WatchSource:0}: Error finding container 44a88b7002386ae057d45d14eb7956f96bbbe51e93dbc550e2114cf295b5ebdf: Status 404 returned error can't find the container with id 44a88b7002386ae057d45d14eb7956f96bbbe51e93dbc550e2114cf295b5ebdf Oct 30 18:32:09.191289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:09.191248 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" event={"ID":"d1c3873b-0c90-4787-99a8-b8ba05807910","Type":"ContainerStarted","Data":"5b9f3f24682b30bec85bbeec83f125b837a4d39442c056978197138b91420e32"} Oct 30 18:32:09.191756 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:09.191315 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" event={"ID":"d1c3873b-0c90-4787-99a8-b8ba05807910","Type":"ContainerStarted","Data":"44a88b7002386ae057d45d14eb7956f96bbbe51e93dbc550e2114cf295b5ebdf"} Oct 30 18:32:09.244612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:09.244566 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" podStartSLOduration=2.2445516420000002 podStartE2EDuration="2.244551642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:09.241106284 +0000 UTC m=+2388.198110121" watchObservedRunningTime="2026-10-30 18:32:09.244551642 +0000 UTC m=+2388.201555478" Oct 30 18:32:10.200294 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:10.200261 4883 generic.go:356] "Generic (PLEG): container finished" podID="d1c3873b-0c90-4787-99a8-b8ba05807910" containerID="5b9f3f24682b30bec85bbeec83f125b837a4d39442c056978197138b91420e32" exitCode=0 Oct 30 18:32:10.200686 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:10.200350 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" event={"ID":"d1c3873b-0c90-4787-99a8-b8ba05807910","Type":"ContainerDied","Data":"5b9f3f24682b30bec85bbeec83f125b837a4d39442c056978197138b91420e32"} Oct 30 18:32:11.421098 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:11.421072 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:11.454820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:11.454788 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-xks-ca\") pod \"d1c3873b-0c90-4787-99a8-b8ba05807910\" (UID: \"d1c3873b-0c90-4787-99a8-b8ba05807910\") " Oct 30 18:32:11.455003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:11.454874 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-openshift-ca\") pod \"d1c3873b-0c90-4787-99a8-b8ba05807910\" (UID: \"d1c3873b-0c90-4787-99a8-b8ba05807910\") " Oct 30 18:32:11.455223 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:11.455202 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c3873b-0c90-4787-99a8-b8ba05807910-kube-api-access-ds56n\" (UniqueName: \"kubernetes.io/projected/d1c3873b-0c90-4787-99a8-b8ba05807910-kube-api-access-ds56n\") pod \"d1c3873b-0c90-4787-99a8-b8ba05807910\" (UID: \"d1c3873b-0c90-4787-99a8-b8ba05807910\") " Oct 30 18:32:11.455273 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:11.455224 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-openshift-ca" pod "d1c3873b-0c90-4787-99a8-b8ba05807910" (UID: "d1c3873b-0c90-4787-99a8-b8ba05807910"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:11.455777 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:11.455755 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:32:11.457353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:11.457325 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c3873b-0c90-4787-99a8-b8ba05807910-kube-api-access-ds56n" pod "d1c3873b-0c90-4787-99a8-b8ba05807910" (UID: "d1c3873b-0c90-4787-99a8-b8ba05807910"). InnerVolumeSpecName "kube-api-access-ds56n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:11.457453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:11.457431 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-xks-ca" pod "d1c3873b-0c90-4787-99a8-b8ba05807910" (UID: "d1c3873b-0c90-4787-99a8-b8ba05807910"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:11.557334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:11.557297 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds56n\" (UniqueName: \"kubernetes.io/projected/d1c3873b-0c90-4787-99a8-b8ba05807910-kube-api-access-ds56n\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:32:11.557334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:11.557331 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c3873b-0c90-4787-99a8-b8ba05807910-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:32:12.217837 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:12.217812 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" Oct 30 18:32:12.218035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:12.217810 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq" event={"ID":"d1c3873b-0c90-4787-99a8-b8ba05807910","Type":"ContainerDied","Data":"44a88b7002386ae057d45d14eb7956f96bbbe51e93dbc550e2114cf295b5ebdf"} Oct 30 18:32:12.218035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:12.217979 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44a88b7002386ae057d45d14eb7956f96bbbe51e93dbc550e2114cf295b5ebdf" Oct 30 18:32:14.557611 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.557574 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw"] Oct 30 18:32:14.559226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.559199 4883 cpu_manager.go:414] "containerMap: removing container" podUID="d1c3873b-0c90-4787-99a8-b8ba05807910" containerName="revoke-keys" Oct 30 18:32:14.559384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.559371 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c3873b-0c90-4787-99a8-b8ba05807910" containerName="revoke-keys" Oct 30 18:32:14.559666 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.559650 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c3873b-0c90-4787-99a8-b8ba05807910" containerName="revoke-keys" Oct 30 18:32:14.566532 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.565066 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.594897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.594867 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e77b1d\"/\"openshift-service-ca.crt\"" Oct 30 18:32:14.594897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.594867 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e77b1d\"/\"auth-test-model-e77b1d-kserve-self-signed-certs\"" Oct 30 18:32:14.594897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.594867 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e77b1d\"/\"kube-root-ca.crt\"" Oct 30 18:32:14.595224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.595052 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e77b1d\"/\"default-dockercfg-9b2s2\"" Oct 30 18:32:14.600422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.600398 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw"] Oct 30 18:32:14.709014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.708981 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-model-cache\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.709214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.709144 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-dshm\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.709214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.709199 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tls-certs\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.709347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.709289 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsbj4\" (UniqueName: \"kubernetes.io/projected/e5d98758-4cc4-49e2-9551-34b1a9be56dc-kube-api-access-bsbj4\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.709488 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.709465 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-home\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.709558 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.709519 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tmp-dir\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.813014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.811045 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbj4\" (UniqueName: \"kubernetes.io/projected/e5d98758-4cc4-49e2-9551-34b1a9be56dc-kube-api-access-bsbj4\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.813014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.811219 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-home\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.813014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.811277 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tmp-dir\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.813014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.811322 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-model-cache\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.813014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.811439 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-dshm\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.813014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.811474 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tls-certs\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.813014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.812321 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tmp-dir\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.813014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.812861 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-home\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.813564 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.813132 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-model-cache\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.841659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.841578 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-dshm\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.856981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.852641 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tls-certs\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:14.896032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:14.872114 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsbj4\" (UniqueName: \"kubernetes.io/projected/e5d98758-4cc4-49e2-9551-34b1a9be56dc-kube-api-access-bsbj4\") pod \"auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:15.182641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:15.182610 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:15.489913 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:15.489886 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw"] Oct 30 18:32:15.490571 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:32:15.490543 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5d98758_4cc4_49e2_9551_34b1a9be56dc.slice/crio-5095bd7aaa0adb53832151a1b7f760ac647a7185c0aa94dd9f9386fd91d3aa6b WatchSource:0}: Error finding container 5095bd7aaa0adb53832151a1b7f760ac647a7185c0aa94dd9f9386fd91d3aa6b: Status 404 returned error can't find the container with id 5095bd7aaa0adb53832151a1b7f760ac647a7185c0aa94dd9f9386fd91d3aa6b Oct 30 18:32:16.256880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:16.256832 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" event={"ID":"e5d98758-4cc4-49e2-9551-34b1a9be56dc","Type":"ContainerStarted","Data":"c4c0db54b5fbd13b11bb562717315cab1f9512523193995adab4f166d40251b8"} Oct 30 18:32:16.257347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:16.256892 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" event={"ID":"e5d98758-4cc4-49e2-9551-34b1a9be56dc","Type":"ContainerStarted","Data":"5095bd7aaa0adb53832151a1b7f760ac647a7185c0aa94dd9f9386fd91d3aa6b"} Oct 30 18:32:16.307718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:16.307660 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podStartSLOduration=2.307646218 podStartE2EDuration="2.307646218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:16.303836003 +0000 UTC m=+2395.260839836" watchObservedRunningTime="2026-10-30 18:32:16.307646218 +0000 UTC m=+2395.264650121" Oct 30 18:32:17.186970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:17.186904 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:19.035976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.035928 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh"] Oct 30 18:32:19.049121 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.049088 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6a443c-hvzkh"] Oct 30 18:32:19.179303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.179266 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32c4e4cf-2711-4319-afa8-acd5ec3a6f4d" path="/var/lib/kubelet/pods/32c4e4cf-2711-4319-afa8-acd5ec3a6f4d/volumes" Oct 30 18:32:19.616580 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.616539 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7"] Oct 30 18:32:19.621981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.621915 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.639751 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.639712 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-50e9fc\"/\"openshift-service-ca.crt\"" Oct 30 18:32:19.642021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.641994 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7"] Oct 30 18:32:19.650888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.650847 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-50e9fc\"/\"kube-root-ca.crt\"" Oct 30 18:32:19.651068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.650846 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-50e9fc\"/\"default-dockercfg-t72dh\"" Oct 30 18:32:19.651227 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.651203 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-50e9fc\"/\"auth-test-model-50e9fc-kserve-self-signed-certs\"" Oct 30 18:32:19.792348 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.792313 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-dshm\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.792546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.792384 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78f62c40-cbfc-42da-8210-5f64075a3e25-tls-certs\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.792601 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.792545 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-tmp-dir\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.792674 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.792654 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhjl\" (UniqueName: \"kubernetes.io/projected/78f62c40-cbfc-42da-8210-5f64075a3e25-kube-api-access-zbhjl\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.792797 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.792782 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-model-cache\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.792898 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.792882 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-home\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.898090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.897994 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-dshm\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.898357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.898339 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78f62c40-cbfc-42da-8210-5f64075a3e25-tls-certs\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.898504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.898492 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-tmp-dir\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.898640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.898624 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhjl\" (UniqueName: \"kubernetes.io/projected/78f62c40-cbfc-42da-8210-5f64075a3e25-kube-api-access-zbhjl\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.898815 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.898803 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-model-cache\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.898990 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.898975 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-home\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.899597 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.899577 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-home\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.900297 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.900278 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-tmp-dir\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.913053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.913014 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-model-cache\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.926627 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.926592 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78f62c40-cbfc-42da-8210-5f64075a3e25-tls-certs\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.934119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.927229 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-dshm\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:19.947096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:19.942922 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhjl\" (UniqueName: \"kubernetes.io/projected/78f62c40-cbfc-42da-8210-5f64075a3e25-kube-api-access-zbhjl\") pod \"auth-test-model-50e9fc-kserve-5f4779568f-4w4t7\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:20.236674 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:20.236596 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:20.586205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:20.586172 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7"] Oct 30 18:32:20.586521 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:32:20.586496 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78f62c40_cbfc_42da_8210_5f64075a3e25.slice/crio-d4d9f9cfdfb99535fd8c03ef14c556415e1938f06c42106cc070b7614e6a49bd WatchSource:0}: Error finding container d4d9f9cfdfb99535fd8c03ef14c556415e1938f06c42106cc070b7614e6a49bd: Status 404 returned error can't find the container with id d4d9f9cfdfb99535fd8c03ef14c556415e1938f06c42106cc070b7614e6a49bd Oct 30 18:32:21.302151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:21.302111 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" event={"ID":"78f62c40-cbfc-42da-8210-5f64075a3e25","Type":"ContainerStarted","Data":"c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691"} Oct 30 18:32:21.302151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:21.302158 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" event={"ID":"78f62c40-cbfc-42da-8210-5f64075a3e25","Type":"ContainerStarted","Data":"d4d9f9cfdfb99535fd8c03ef14c556415e1938f06c42106cc070b7614e6a49bd"} Oct 30 18:32:21.352471 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:21.352417 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podStartSLOduration=2.352403686 podStartE2EDuration="2.352403686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:21.348608443 +0000 UTC m=+2400.305612280" watchObservedRunningTime="2026-10-30 18:32:21.352403686 +0000 UTC m=+2400.309407523" Oct 30 18:32:21.848045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:21.848016 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:32:21.857293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:21.857264 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:32:21.864213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:21.864188 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:32:21.872376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:21.872358 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:32:22.237716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:22.237643 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:22.766291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:22.766255 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5"] Oct 30 18:32:22.770624 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:22.770602 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:22.788054 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:22.788030 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:32:22.801447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:22.801419 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:32:22.801579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:22.801474 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:32:22.801579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:22.801551 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:32:22.807198 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:22.807163 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5"] Oct 30 18:32:22.944161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:22.944121 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d21505d3-c5rl5\" (UID: \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:22.944332 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:22.944280 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d21505d3-c5rl5\" (UID: \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:22.944443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:22.944428 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drj5j\" (UniqueName: \"kubernetes.io/projected/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-kube-api-access-drj5j\") pod \"maas-api-revoke-sub-d21505d3-c5rl5\" (UID: \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:23.047312 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:23.047217 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d21505d3-c5rl5\" (UID: \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:23.047487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:23.047420 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drj5j\" (UniqueName: \"kubernetes.io/projected/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-kube-api-access-drj5j\") pod \"maas-api-revoke-sub-d21505d3-c5rl5\" (UID: \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:23.047590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:23.047573 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d21505d3-c5rl5\" (UID: \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:23.050876 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:23.049225 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d21505d3-c5rl5\" (UID: \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:23.058967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:23.058919 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d21505d3-c5rl5\" (UID: \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:23.073463 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:23.073433 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drj5j\" (UniqueName: \"kubernetes.io/projected/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-kube-api-access-drj5j\") pod \"maas-api-revoke-sub-d21505d3-c5rl5\" (UID: \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:23.084305 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:23.084285 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:23.434496 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:23.434462 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5"] Oct 30 18:32:23.434496 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:32:23.434483 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2ecef6_4d24_4410_b822_ce7a03f6b2e0.slice/crio-97e8c5a673af149c91997a753f237aa715ab11226f99e82ec3685ba4c1966b18 WatchSource:0}: Error finding container 97e8c5a673af149c91997a753f237aa715ab11226f99e82ec3685ba4c1966b18: Status 404 returned error can't find the container with id 97e8c5a673af149c91997a753f237aa715ab11226f99e82ec3685ba4c1966b18 Oct 30 18:32:24.328975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:24.328920 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" event={"ID":"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0","Type":"ContainerStarted","Data":"8e63d2e7f63675b60692c1063bd49de20437dbe1fbf700368fa63ed7f8759725"} Oct 30 18:32:24.329433 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:24.328996 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" event={"ID":"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0","Type":"ContainerStarted","Data":"97e8c5a673af149c91997a753f237aa715ab11226f99e82ec3685ba4c1966b18"} Oct 30 18:32:24.375862 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:24.375804 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" podStartSLOduration=2.375789743 podStartE2EDuration="2.375789743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:24.371834853 +0000 UTC m=+2403.328838689" watchObservedRunningTime="2026-10-30 18:32:24.375789743 +0000 UTC m=+2403.332793578" Oct 30 18:32:25.183032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:25.183001 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:25.195942 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:25.195916 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:25.341145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:25.341094 4883 generic.go:356] "Generic (PLEG): container finished" podID="cc2ecef6-4d24-4410-b822-ce7a03f6b2e0" containerID="8e63d2e7f63675b60692c1063bd49de20437dbe1fbf700368fa63ed7f8759725" exitCode=0 Oct 30 18:32:25.341592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:25.341177 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" event={"ID":"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0","Type":"ContainerDied","Data":"8e63d2e7f63675b60692c1063bd49de20437dbe1fbf700368fa63ed7f8759725"} Oct 30 18:32:25.353005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:25.352982 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:32:26.034621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:26.034587 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2"] Oct 30 18:32:26.047207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:26.047171 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f4b644-hf7r2"] Oct 30 18:32:26.587040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:26.587011 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:26.612430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:26.612346 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-xks-ca\") pod \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\" (UID: \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\") " Oct 30 18:32:26.612574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:26.612460 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-openshift-ca\") pod \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\" (UID: \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\") " Oct 30 18:32:26.612690 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:26.612673 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-kube-api-access-drj5j\" (UniqueName: \"kubernetes.io/projected/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-kube-api-access-drj5j\") pod \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\" (UID: \"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0\") " Oct 30 18:32:26.612837 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:26.612816 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-openshift-ca" pod "cc2ecef6-4d24-4410-b822-ce7a03f6b2e0" (UID: "cc2ecef6-4d24-4410-b822-ce7a03f6b2e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:26.613267 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:26.613247 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:32:26.614547 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:26.614516 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-xks-ca" pod "cc2ecef6-4d24-4410-b822-ce7a03f6b2e0" (UID: "cc2ecef6-4d24-4410-b822-ce7a03f6b2e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:26.614649 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:26.614590 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-kube-api-access-drj5j" pod "cc2ecef6-4d24-4410-b822-ce7a03f6b2e0" (UID: "cc2ecef6-4d24-4410-b822-ce7a03f6b2e0"). InnerVolumeSpecName "kube-api-access-drj5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:26.714440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:26.714382 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:32:26.714440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:26.714430 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drj5j\" (UniqueName: \"kubernetes.io/projected/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0-kube-api-access-drj5j\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:32:27.180311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:27.180275 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f06e8d-eab3-445d-98f5-a300fcddb2b0" path="/var/lib/kubelet/pods/98f06e8d-eab3-445d-98f5-a300fcddb2b0/volumes" Oct 30 18:32:27.360155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:27.360121 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" Oct 30 18:32:27.360352 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:27.360136 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5" event={"ID":"cc2ecef6-4d24-4410-b822-ce7a03f6b2e0","Type":"ContainerDied","Data":"97e8c5a673af149c91997a753f237aa715ab11226f99e82ec3685ba4c1966b18"} Oct 30 18:32:27.360352 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:27.360204 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97e8c5a673af149c91997a753f237aa715ab11226f99e82ec3685ba4c1966b18" Oct 30 18:32:30.084430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:30.084397 4883 scope.go:122] "RemoveContainer" containerID="5f75582f2bee6b804d2665a25d916e25cd09b5ee4f7d8429ab913f642ef8d41d" Oct 30 18:32:30.095811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:30.095781 4883 scope.go:122] "RemoveContainer" containerID="4f4f9a7090575afb33494fc62677bf8ec16bb924fd7ec55aeeb35f01bca61e69" Oct 30 18:32:30.109778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:30.109749 4883 scope.go:122] "RemoveContainer" containerID="03f6ef8b2b5fbf7134e197ef5413aa5bd9f1376a11ae941522098f58adb06a7e" Oct 30 18:32:30.134101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:30.134074 4883 scope.go:122] "RemoveContainer" containerID="37b6a2b72159da4d068431f4618e761d3babb33325bdc348496643ffdc685565" Oct 30 18:32:30.146441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:30.146421 4883 scope.go:122] "RemoveContainer" containerID="950127894fe22f35fa3f6b81983def9c19b0ef6b3ae85354782ecd144b1946e3" Oct 30 18:32:30.159684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:30.159667 4883 scope.go:122] "RemoveContainer" containerID="2396dd330476af89031750aca863b6a8689d1133d5dcc8a2c60e9b3b49a9008e" Oct 30 18:32:30.237018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:30.236993 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:30.249582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:30.249558 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:30.395586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:30.395556 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:32:32.033007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:32.032971 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj"] Oct 30 18:32:32.046487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:32.046460 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d9321d-8nmmj"] Oct 30 18:32:33.179567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:33.179537 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706d9904-f6e8-487b-92e9-ba25aa0bedac" path="/var/lib/kubelet/pods/706d9904-f6e8-487b-92e9-ba25aa0bedac/volumes" Oct 30 18:32:35.081364 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:32:35.081326 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:32:42.042103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:42.042063 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s"] Oct 30 18:32:42.054153 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:42.054112 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103c5bee-xpb6s"] Oct 30 18:32:43.033614 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:43.033582 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7"] Oct 30 18:32:43.046535 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:43.046504 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a8d4878-jqzv7"] Oct 30 18:32:43.188799 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:43.188753 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="891c9dfe-3da4-480d-867d-54295c2fa8c6" path="/var/lib/kubelet/pods/891c9dfe-3da4-480d-867d-54295c2fa8c6/volumes" Oct 30 18:32:43.189312 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:43.189289 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21df428-2c21-4e18-b29e-2e7deca7274e" path="/var/lib/kubelet/pods/d21df428-2c21-4e18-b29e-2e7deca7274e/volumes" Oct 30 18:32:54.043190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:54.043150 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp"] Oct 30 18:32:54.055472 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:54.055424 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79f463ca-dkdxp"] Oct 30 18:32:54.059461 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:54.059435 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq"] Oct 30 18:32:54.066849 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:54.066823 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-77db82-rjkxq"] Oct 30 18:32:55.179496 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:55.179464 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e18119b-d51e-4470-8a2b-285c1bd0b905" path="/var/lib/kubelet/pods/0e18119b-d51e-4470-8a2b-285c1bd0b905/volumes" Oct 30 18:32:55.179967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:55.179951 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c3873b-0c90-4787-99a8-b8ba05807910" path="/var/lib/kubelet/pods/d1c3873b-0c90-4787-99a8-b8ba05807910/volumes" Oct 30 18:32:56.076683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:56.076642 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q"] Oct 30 18:32:56.081654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:56.081602 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b34c5be7-hgz2q"] Oct 30 18:32:57.178903 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:32:57.178868 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f14729-a34e-477b-9a84-4fe5af1175fa" path="/var/lib/kubelet/pods/59f14729-a34e-477b-9a84-4fe5af1175fa/volumes" Oct 30 18:33:03.042463 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:03.042425 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78"] Oct 30 18:33:03.046404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:03.046365 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee130c78-qts78"] Oct 30 18:33:03.180638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:03.180601 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775f2dca-b4b5-438d-a835-93bef761df1a" path="/var/lib/kubelet/pods/775f2dca-b4b5-438d-a835-93bef761df1a/volumes" Oct 30 18:33:05.081731 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:33:05.081694 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:33:18.050582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:18.050548 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7"] Oct 30 18:33:18.063754 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:18.063723 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-263d960f-bnrx7"] Oct 30 18:33:19.179870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:19.179838 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143c1190-bf70-4c79-918f-1d4c891c35db" path="/var/lib/kubelet/pods/143c1190-bf70-4c79-918f-1d4c891c35db/volumes" Oct 30 18:33:22.358226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.357701 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.391134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.391027 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.419199 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.418059 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.584637 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.584580 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.653165 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.653039 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.653858 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.653459 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.729106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.729037 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.730373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.730338 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.731095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.730573 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.841524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.841470 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:22.859287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.859245 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:33:22.986014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.957078 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:22.986014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:22.957164 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.070960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.067089 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:23.070960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.067155 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.156183 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.155029 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 18:33:23.169047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.168019 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.331029 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.321357 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:33:23.366000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.358470 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:33:23.366000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.358779 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.366000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.359455 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:33:23.402963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.393009 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.402963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.393724 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:33:23.434557 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.431758 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.434557 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.432228 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:33:23.613015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.589110 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.688547 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.688513 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:33:23.713170 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.689744 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.770979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.689795 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.770979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.718225 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:33:23.770979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.718285 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:33:23.770979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.741021 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.770979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.741618 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:33:23.770979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.742649 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.770979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.742950 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.770979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.743406 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:33:23.770979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.743454 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:33:23.888565 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.856746 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.888565 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.880924 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:23.922347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.920671 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": context deadline exceeded" Oct 30 18:33:23.922347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.921153 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:33:23.922347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:23.921236 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:33:24.180092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.159466 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 18:33:24.180092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.176324 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:33:24.180092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.179702 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:24.305800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.296205 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:33:24.337965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.333251 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:24.337965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.334264 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:33:24.381973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.364724 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:24.381973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.365978 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": context deadline exceeded" Oct 30 18:33:24.405972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.391421 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:33:24.405972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.393756 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:24.426960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.424471 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:24.461021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.445044 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:24.723067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.698433 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:33:24.799442 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.746286 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:24.800143 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.746336 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:24.828598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.746367 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:24.828895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.748254 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:33:24.829829 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.748284 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:33:24.894958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.869663 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.138389 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.886460 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.196182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:24.925833 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.208817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.138666 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:33:25.208817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.187834 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.208817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.187905 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.375979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.363373 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:33:25.375979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.366723 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.375979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.367999 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.427032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.396276 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.498079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.498037 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.625072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.625021 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.785441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.785334 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.803610 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.803561 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:25.820096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.820038 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:33:25.823869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.823709 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:33:25.832215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.832162 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:33:25.840419 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:25.840375 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:26.168962 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.145516 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:26.303653 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.198450 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:26.344740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.198507 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": context deadline exceeded" Oct 30 18:33:26.344740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.198545 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/health\": context deadline exceeded" Oct 30 18:33:26.344740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.209325 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:26.558022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.554218 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:26.651011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.585109 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:33:26.693998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.209360 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:33:26.693998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.615691 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:26.693998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.684722 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 30 18:33:26.756959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.615812 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:26.756959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.629038 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:33:26.857021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.830383 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.013963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.847506 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.053694 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.900708 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:33:27.053694 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.919463 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.053694 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:26.919571 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:33:27.061693 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.061159 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.095971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.083657 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:33:27.420045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.402443 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": context deadline exceeded" Oct 30 18:33:27.489969 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.411435 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 18:33:27.520244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.520195 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:33:27.520912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.520869 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:33:27.595986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.590476 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:33:27.683661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.683423 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.691316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.691274 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:33:27.720523 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.692859 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.760506 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.760460 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": context deadline exceeded" Oct 30 18:33:27.783652 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.783602 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.927978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.898088 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:27.984427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:27.984309 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.029457 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.029412 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:33:28.041094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.041048 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.048965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.041679 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:33:28.066970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.050655 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:33:28.167972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.166025 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:33:28.392957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.391324 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.450400 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.450259 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded" Oct 30 18:33:28.474961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.463110 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:28.474961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.463189 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.484186 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.484128 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.484769 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.484734 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.520643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.520601 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.528074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.528029 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.591188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.590649 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:33:28.694125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.694019 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.695023 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.694991 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:33:28.726400 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.726349 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.812105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.812061 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:28.858758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.812383 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": context deadline exceeded" Oct 30 18:33:28.986217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:28.986090 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.035113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.035053 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.035760 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.035715 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:33:29.051671 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.051626 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.052519 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.051914 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.090269 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.090223 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.122093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.122049 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.181959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.170164 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.310473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.310356 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.392125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.392077 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.445128 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.445074 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.487226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.487166 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.488354 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.488312 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.521091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.521037 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.528524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.528473 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.591752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.591651 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.696104 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.696056 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:33:29.697027 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.696996 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:33:29.730099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.730037 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.754075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.754028 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.859103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.859050 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:29.860401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.860373 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": context deadline exceeded" Oct 30 18:33:29.975088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.975040 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Oct 30 18:33:29.992469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:29.992421 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:33:30.040119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.040061 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.056091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.056044 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.056721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.056688 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.092117 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.092070 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.172452 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.172351 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:33:30.256965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.247846 4883 scope.go:122] "RemoveContainer" containerID="bfba356e00942b0cdfac9b07cae4d0602dfb05169495bdbea4412fba1fad9c77" Oct 30 18:33:30.299633 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.299206 4883 scope.go:122] "RemoveContainer" containerID="2fff9c25b4160383bd8543bbec346bb96b439b650a89bd46eb90e30ebeb78ff2" Oct 30 18:33:30.354189 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.354158 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:33:30.375418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.375385 4883 scope.go:122] "RemoveContainer" containerID="fbc0d53ee178b43fa45be3d09a4cff7174613c5b811929526a266c0548048068" Oct 30 18:33:30.493085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.491237 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.493085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.491618 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Oct 30 18:33:30.527998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.523888 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.572094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.572050 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.619498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.572344 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:33:30.619807 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.572980 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.620987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.592523 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.704987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.697578 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.704987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.698045 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:33:30.790031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.784064 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.810110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.810075 4883 scope.go:122] "RemoveContainer" containerID="f10e81b745834f51445d17d00d5ae5e520045d9b0314608f50664684a2a0ae0e" Oct 30 18:33:30.888064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.888020 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.931096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.931053 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:30.997457 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:30.997418 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:33:31.037773 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.037624 4883 scope.go:122] "RemoveContainer" containerID="7a49af92cd71a79abcfb320b6fb50a0f7f627e7542aa760a7b68c8a89905fc33" Oct 30 18:33:31.063015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.054746 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs"] Oct 30 18:33:31.063015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.054841 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:33:31.063015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.058048 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3691a87-mnzgs"] Oct 30 18:33:31.063015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.058064 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.063015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.058303 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:33:31.117958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.117029 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.177448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.177401 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.206453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.206412 4883 scope.go:122] "RemoveContainer" containerID="a5879f9599d141a7855ce2a0b67e80979d10a860f4de4f68b9ab36c7964e4ba2" Oct 30 18:33:31.474101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.432325 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:33:31.517984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.514603 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.517984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.501906 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.517984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.496486 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": context deadline exceeded" Oct 30 18:33:31.670029 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.668440 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.681543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.680692 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:33:31.730060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.708285 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:33:31.730060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.722725 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.730060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.729247 4883 scope.go:122] "RemoveContainer" containerID="48f6de5dc8748b1b9cc978f79d28dac27d8eb98e561ad1142878dcf358b3add1" Oct 30 18:33:31.760547 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.760475 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.784644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.784596 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee1c9c5-22ce-49b6-9db6-61431873f1ab" path="/var/lib/kubelet/pods/fee1c9c5-22ce-49b6-9db6-61431873f1ab/volumes" Oct 30 18:33:31.785375 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.784794 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:31.823130 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.822688 4883 scope.go:122] "RemoveContainer" containerID="ef980095d07dc0a46adba53c5825f5554e6fb9454598723822ec19b7d0d2c9ca" Oct 30 18:33:31.882198 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.882163 4883 scope.go:122] "RemoveContainer" containerID="7f31a1f5c3b74225444ab8ea32a3bf3a450dfae05a3ba51d3a3fa839bb2d5964" Oct 30 18:33:31.886913 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.886887 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:33:31.913278 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.912897 4883 scope.go:122] "RemoveContainer" containerID="e551080a7705efa936a32a2061887d67e5ed33609eccaff99aa00acffd41556e" Oct 30 18:33:31.938309 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:31.936568 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:32.080747 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.080692 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:33:32.106084 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.106051 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:33:32.108147 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.108114 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:33:32.218964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.211458 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:33:32.342745 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.340148 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:33:32.625216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.608386 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:32.646709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.646677 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:33:32.654830 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.654802 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:33:32.662089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.662022 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:33:32.664580 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.661405 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:32.693568 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.693534 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:33:32.709180 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.709154 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:33:32.710040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.709995 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:33:32.716474 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.715495 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:33:32.717909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.717828 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:33:32.728441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.718755 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:33:32.728441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.721757 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:33:32.743017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.742302 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:33:32.743017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:32.742800 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:33:34.037156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:34.037122 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g"] Oct 30 18:33:34.050572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:34.050542 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe9ff29-44p4g"] Oct 30 18:33:35.089260 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:33:35.089217 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:33:35.181991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:35.181301 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0079de08-730b-4941-b639-394b463d5d3a" path="/var/lib/kubelet/pods/0079de08-730b-4941-b639-394b463d5d3a/volumes" Oct 30 18:33:38.775010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:38.774967 4883 generic.go:356] "Generic (PLEG): container finished" podID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerID="42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3" exitCode=0 Oct 30 18:33:38.775484 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:38.775038 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" event={"ID":"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d","Type":"ContainerDied","Data":"42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3"} Oct 30 18:33:38.775557 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:38.775532 4883 scope.go:122] "RemoveContainer" containerID="42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3" Oct 30 18:33:38.777326 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:38.777295 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:38.777409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:38.777367 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:38.777409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:38.777400 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:39.797874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:39.797841 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" event={"ID":"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d","Type":"ContainerStarted","Data":"a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920"} Oct 30 18:33:40.040479 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:40.040431 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg"] Oct 30 18:33:40.040903 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:40.040502 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d636b736-wjrmg"] Oct 30 18:33:40.744897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:40.744854 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:33:40.745106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:40.745072 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 18:33:40.745216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:40.745166 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 18:33:41.032987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.032868 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg"] Oct 30 18:33:41.045707 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.045679 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b8b7dd-ddwjg"] Oct 30 18:33:41.224519 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.224477 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed55dba-0cb9-45c9-970e-6607ba087a21" path="/var/lib/kubelet/pods/2ed55dba-0cb9-45c9-970e-6607ba087a21/volumes" Oct 30 18:33:41.225532 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.225505 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2555b8c-3a91-4de0-892f-af62a83a2b5c" path="/var/lib/kubelet/pods/c2555b8c-3a91-4de0-892f-af62a83a2b5c/volumes" Oct 30 18:33:41.744967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.744895 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 18:33:41.745153 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.744994 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 18:33:41.838386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.838346 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf"] Oct 30 18:33:41.839067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.839037 4883 cpu_manager.go:414] "containerMap: removing container" podUID="cc2ecef6-4d24-4410-b822-ce7a03f6b2e0" containerName="revoke-keys" Oct 30 18:33:41.839067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.839063 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2ecef6-4d24-4410-b822-ce7a03f6b2e0" containerName="revoke-keys" Oct 30 18:33:41.839332 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.839313 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2ecef6-4d24-4410-b822-ce7a03f6b2e0" containerName="revoke-keys" Oct 30 18:33:41.842768 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.842746 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:41.859848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.859823 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b00d69-openshift-default-dockercfg-dp9gr\"" Oct 30 18:33:41.899534 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.899500 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:41.899738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.899562 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srnm\" (UniqueName: \"kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-kube-api-access-8srnm\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:41.899738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.899691 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:41.899871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.899750 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:41.899871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.899796 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:41.899871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.899851 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:41.900070 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.899915 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:41.900070 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.899972 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b558af39-00d4-41f8-b004-29437a7d270a-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:41.900070 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.900011 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b558af39-00d4-41f8-b004-29437a7d270a-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:41.903535 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:41.903505 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf"] Oct 30 18:33:42.000841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.000754 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.000841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.000829 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.001115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.000894 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.001115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.000926 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b558af39-00d4-41f8-b004-29437a7d270a-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.001115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.000972 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b558af39-00d4-41f8-b004-29437a7d270a-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.001115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.001034 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.001115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.001096 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srnm\" (UniqueName: \"kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-kube-api-access-8srnm\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.001400 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.001342 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.001400 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.001371 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.001506 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.001411 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.001683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.001651 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.001761 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.001749 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.001827 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.001784 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b558af39-00d4-41f8-b004-29437a7d270a-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.002065 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.002043 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.003329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.003308 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.003604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.003584 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b558af39-00d4-41f8-b004-29437a7d270a-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.040963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.040922 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.046570 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.046541 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srnm\" (UniqueName: \"kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-kube-api-access-8srnm\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.157717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.157681 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:42.525757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.525726 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf"] Oct 30 18:33:42.529885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.529840 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:42.530015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.529926 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:42.530015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.529994 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:42.745212 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.745175 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 18:33:42.745386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.745235 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 18:33:42.839291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.839197 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" event={"ID":"b558af39-00d4-41f8-b004-29437a7d270a","Type":"ContainerStarted","Data":"99a2827c7204af3f2daa9476d87a66a71cdb5e1f3eed3f5773fa0b736527231e"} Oct 30 18:33:42.839475 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:42.839300 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" event={"ID":"b558af39-00d4-41f8-b004-29437a7d270a","Type":"ContainerStarted","Data":"098f8adad0c738130188f52402e544e5f4d78b6567bd748c017cb9cf2ea9db92"} Oct 30 18:33:43.162905 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:43.158502 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:43.162905 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:43.159669 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:33:43.162905 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:43.159720 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" podUID="b558af39-00d4-41f8-b004-29437a7d270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:33:43.745599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:43.745566 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 18:33:43.745802 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:43.745620 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 18:33:44.158336 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:44.158298 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:33:44.158524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:44.158358 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" podUID="b558af39-00d4-41f8-b004-29437a7d270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:33:44.605151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:44.603728 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" podStartSLOduration=3.603704862 podStartE2EDuration="3.603704862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:42.902311495 +0000 UTC m=+2481.859315332" watchObservedRunningTime="2026-10-30 18:33:44.603704862 +0000 UTC m=+2483.560708700" Oct 30 18:33:44.648589 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:44.648551 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78"] Oct 30 18:33:44.649131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:44.649093 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" podUID="48259124-69a9-4121-91cb-337d2724acb9" containerName="istio-proxy" containerID="cri-o://24f4d64d7f2f2025df2aeaeb7e5a7f5bf5eb2d3824429a40e56b78f73e1e8a16" gracePeriod=30 Oct 30 18:33:44.654675 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:44.654656 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f"] Oct 30 18:33:44.658700 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:44.658543 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" podUID="91bacf95-f115-438a-a3cf-e7c753ec1a82" containerName="istio-proxy" containerID="cri-o://424530e746905bfbcfe817a84327f9373478767c3135b18159b547e0aafd903b" gracePeriod=30 Oct 30 18:33:44.740307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:44.739637 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw"] Oct 30 18:33:44.740307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:44.739889 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" podUID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" containerName="istio-proxy" containerID="cri-o://ae60207c70c62d74f097143a236f5e58f5fd2f2b11a0196ed3493eb3fa656e99" gracePeriod=30 Oct 30 18:33:44.796976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:44.775619 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Oct 30 18:33:44.796976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:44.775680 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Oct 30 18:33:45.125228 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.124771 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx"] Oct 30 18:33:45.150687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.150656 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx"] Oct 30 18:33:45.150885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.150780 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.159422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.159374 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:33:45.159595 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.159456 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" podUID="b558af39-00d4-41f8-b004-29437a7d270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:33:45.209520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.208406 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-3b8bf3-openshift-default-dockercfg-256pt\"" Oct 30 18:33:45.249956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.245254 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-podinfo\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.249956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.245310 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-envoy\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.249956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.245347 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecdc930-9541-4a66-9967-4fb76ca616d9-istiod-ca-cert\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.249956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.245432 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-data\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.249956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.245561 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-token\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.249956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.245634 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-certs\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.249956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.245687 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-credential-socket\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.249956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.245745 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-socket\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.249956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.245891 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxhc\" (UniqueName: \"kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-kube-api-access-sdxhc\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.327128 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.327089 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr"] Oct 30 18:33:45.331321 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.331297 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:45.348126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.347259 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecdc930-9541-4a66-9967-4fb76ca616d9-istiod-ca-cert\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.348126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.347357 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-data\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.348126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.347492 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-token\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.348126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.347560 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-certs\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.348126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.347579 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:33:45.348126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.347606 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-credential-socket\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.348126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.347665 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-socket\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.348126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.347745 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:33:45.348126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.347817 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxhc\" (UniqueName: \"kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-kube-api-access-sdxhc\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.348716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.348242 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-credential-socket\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.348716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.348481 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-socket\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.348716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.348634 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-podinfo\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.348716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.348675 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-envoy\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.349404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.349379 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecdc930-9541-4a66-9967-4fb76ca616d9-istiod-ca-cert\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.349530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.349498 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-certs\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.350068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.349808 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-data\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.351231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.351206 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr"] Oct 30 18:33:45.353315 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.353291 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-podinfo\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.361347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.357351 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-envoy\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.407228 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.407154 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:33:45.407979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.407425 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:33:45.407979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.407909 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-token\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.414641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.414614 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxhc\" (UniqueName: \"kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-kube-api-access-sdxhc\") pod \"e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.449955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.449849 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr\" (UID: \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:45.450128 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.449982 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn95r\" (UniqueName: \"kubernetes.io/projected/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-kube-api-access-vn95r\") pod \"maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr\" (UID: \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:45.451035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.450205 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr\" (UID: \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:45.540646 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.540604 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:45.552607 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.552560 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr\" (UID: \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:45.552767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.552654 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn95r\" (UniqueName: \"kubernetes.io/projected/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-kube-api-access-vn95r\") pod \"maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr\" (UID: \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:45.553067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.553038 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr\" (UID: \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:45.586015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.585980 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr\" (UID: \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:45.621604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.589308 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr\" (UID: \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:45.636224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.635775 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn95r\" (UniqueName: \"kubernetes.io/projected/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-kube-api-access-vn95r\") pod \"maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr\" (UID: \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:45.645230 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.645199 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:45.756914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.753176 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:33:45.756914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.754702 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:33:45.756914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:45.756236 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:33:46.006513 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:46.006436 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx"] Oct 30 18:33:46.012972 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:33:46.012358 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecdc930_9541_4a66_9967_4fb76ca616d9.slice/crio-63bcd9550e4decbee141b558f4649910b24beec1725c52c8a2e6705bc8321a22 WatchSource:0}: Error finding container 63bcd9550e4decbee141b558f4649910b24beec1725c52c8a2e6705bc8321a22: Status 404 returned error can't find the container with id 63bcd9550e4decbee141b558f4649910b24beec1725c52c8a2e6705bc8321a22 Oct 30 18:33:46.021618 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:46.021579 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:46.021730 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:46.021674 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:46.021730 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:46.021722 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:46.061286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:46.061260 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr"] Oct 30 18:33:46.109212 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:46.109088 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" event={"ID":"3ecdc930-9541-4a66-9967-4fb76ca616d9","Type":"ContainerStarted","Data":"63bcd9550e4decbee141b558f4649910b24beec1725c52c8a2e6705bc8321a22"} Oct 30 18:33:46.159689 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:46.158400 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:33:46.159689 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:46.158454 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" podUID="b558af39-00d4-41f8-b004-29437a7d270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:33:47.127991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:47.127641 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" event={"ID":"3ecdc930-9541-4a66-9967-4fb76ca616d9","Type":"ContainerStarted","Data":"f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753"} Oct 30 18:33:47.130222 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:47.130193 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" event={"ID":"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3","Type":"ContainerStarted","Data":"6b58cdc653507de7ad942051a7b808df10078c3752e1f251108d70572189db49"} Oct 30 18:33:47.130395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:47.130240 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" event={"ID":"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3","Type":"ContainerStarted","Data":"d5fee5e7738420b8e00d51925bb0a10dad5b38fdf426a9612be31c20b60101c3"} Oct 30 18:33:47.158610 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:47.158084 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:33:47.158610 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:47.158129 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" podUID="b558af39-00d4-41f8-b004-29437a7d270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:33:47.241321 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:47.240668 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" podStartSLOduration=3.24064883 podStartE2EDuration="3.24064883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:47.195794707 +0000 UTC m=+2486.152798544" watchObservedRunningTime="2026-10-30 18:33:47.24064883 +0000 UTC m=+2486.197652679" Oct 30 18:33:47.242141 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:47.242088 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" podStartSLOduration=2.242072643 podStartE2EDuration="2.242072643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:47.240374895 +0000 UTC m=+2486.197378732" watchObservedRunningTime="2026-10-30 18:33:47.242072643 +0000 UTC m=+2486.199076482" Oct 30 18:33:47.546481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:47.546381 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:47.547555 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:47.547527 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:33:47.547671 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:47.547578 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" podUID="3ecdc930-9541-4a66-9967-4fb76ca616d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 18:33:48.138656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:48.138615 4883 generic.go:356] "Generic (PLEG): container finished" podID="c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3" containerID="6b58cdc653507de7ad942051a7b808df10078c3752e1f251108d70572189db49" exitCode=0 Oct 30 18:33:48.139139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:48.138703 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" event={"ID":"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3","Type":"ContainerDied","Data":"6b58cdc653507de7ad942051a7b808df10078c3752e1f251108d70572189db49"} Oct 30 18:33:48.144695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:48.144667 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:33:48.144823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:48.144721 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" podUID="48259124-69a9-4121-91cb-337d2724acb9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:33:48.158850 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:48.158824 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:33:48.158980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:48.158864 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" podUID="b558af39-00d4-41f8-b004-29437a7d270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:33:48.546540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:48.546448 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:33:48.546540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:48.546511 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" podUID="3ecdc930-9541-4a66-9967-4fb76ca616d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 18:33:49.159101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.158997 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:33:49.159101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.159057 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" podUID="b558af39-00d4-41f8-b004-29437a7d270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Oct 30 18:33:49.220602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.220412 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9"] Oct 30 18:33:49.224883 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.224847 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.260990 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.254659 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9"] Oct 30 18:33:49.284985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.262426 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-e77b1d-openshift-default-dockercfg-k7wxj\"" Oct 30 18:33:49.403821 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.403707 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qrs6\" (UniqueName: \"kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-kube-api-access-4qrs6\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.415975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.412084 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.415975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.408862 4883 patch_prober.go:29] interesting pod/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:33:49.415975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.412182 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" podUID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:33:49.415975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.412271 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-envoy\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.415975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.412341 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-credential-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.415975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.412467 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-token\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.415975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.412648 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-certs\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.415975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.412701 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-data\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.415975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.412728 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istiod-ca-cert\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.415975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.412760 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-podinfo\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.416540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.416520 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:49.513473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.513438 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-kube-api-access-vn95r\" (UniqueName: \"kubernetes.io/projected/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-kube-api-access-vn95r\") pod \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\" (UID: \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\") " Oct 30 18:33:49.513676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.513660 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-openshift-ca\") pod \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\" (UID: \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\") " Oct 30 18:33:49.513827 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.513757 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-xks-ca\") pod \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\" (UID: \"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3\") " Oct 30 18:33:49.514078 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.514034 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-certs\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.514238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.514095 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-data\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.514238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.514122 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istiod-ca-cert\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.514238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.514155 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-podinfo\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.514397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.514234 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qrs6\" (UniqueName: \"kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-kube-api-access-4qrs6\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.514397 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.514295 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.514511 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.514413 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-envoy\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.514511 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.514484 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-credential-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.514617 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.514574 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-token\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.517021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.515615 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.517021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.516637 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-openshift-ca" pod "c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3" (UID: "c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:49.517021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.516835 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-credential-socket\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.517276 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.517031 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-data\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.517429 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.517396 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-certs\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.518517 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.518462 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istiod-ca-cert\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.534048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.523632 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-envoy\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.534048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.526354 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-kube-api-access-vn95r" pod "c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3" (UID: "c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3"). InnerVolumeSpecName "kube-api-access-vn95r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:49.534048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.530404 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-podinfo\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.534819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.534793 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-xks-ca" pod "c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3" (UID: "c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:49.542394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.542342 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-token\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.544404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.544233 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qrs6\" (UniqueName: \"kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-kube-api-access-4qrs6\") pod \"e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.547140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.547116 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:33:49.547353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.547310 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" podUID="3ecdc930-9541-4a66-9967-4fb76ca616d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 18:33:49.616241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.616212 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:49.616407 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.616397 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:49.616464 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.616456 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn95r\" (UniqueName: \"kubernetes.io/projected/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3-kube-api-access-vn95r\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:49.739413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.733441 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq"] Oct 30 18:33:49.739413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.734389 4883 cpu_manager.go:414] "containerMap: removing container" podUID="c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3" containerName="revoke-keys" Oct 30 18:33:49.739413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.734407 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3" containerName="revoke-keys" Oct 30 18:33:49.739413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.734560 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3" containerName="revoke-keys" Oct 30 18:33:49.739413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.737978 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.776967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.765487 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq"] Oct 30 18:33:49.776967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.766555 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ac632e-openshift-default-dockercfg-vbqgb\"" Oct 30 18:33:49.830793 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.830757 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-certs\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.832132 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.831091 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-data\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.832132 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.831182 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-token\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.833025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.831220 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-envoy\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.833025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.832481 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-podinfo\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.833025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.832586 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-socket\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.833025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.832625 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-credential-socket\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.833025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.832721 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istiod-ca-cert\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.833025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.832746 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx47s\" (UniqueName: \"kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-kube-api-access-jx47s\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.842988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.842955 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:49.934151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.934063 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-certs\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.934151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.934121 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-data\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.934428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.934204 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-token\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.934428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.934242 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-envoy\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.934428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.934306 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-podinfo\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.934428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.934395 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-socket\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.934673 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.934433 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-credential-socket\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.934673 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.934527 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istiod-ca-cert\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.934673 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.934556 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx47s\" (UniqueName: \"kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-kube-api-access-jx47s\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.938075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.935879 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-credential-socket\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.938075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.937166 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-data\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.938075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.937185 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-socket\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.938075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.937759 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istiod-ca-cert\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.938075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.938041 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-certs\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.939566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.939516 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-podinfo\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:49.940211 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.940188 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-envoy\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:50.004080 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:49.979122 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx47s\" (UniqueName: \"kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-kube-api-access-jx47s\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:50.006947 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.006372 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-token\") pod \"e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:50.156696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.156489 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" Oct 30 18:33:50.156696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.156592 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr" event={"ID":"c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3","Type":"ContainerDied","Data":"d5fee5e7738420b8e00d51925bb0a10dad5b38fdf426a9612be31c20b60101c3"} Oct 30 18:33:50.156696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.156635 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5fee5e7738420b8e00d51925bb0a10dad5b38fdf426a9612be31c20b60101c3" Oct 30 18:33:50.161183 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.160952 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:50.218024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.193567 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:50.218024 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.207029 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:50.220446 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.220371 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:33:50.234047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.233748 4883 generic.go:356] "Generic (PLEG): container finished" podID="91bacf95-f115-438a-a3cf-e7c753ec1a82" containerID="424530e746905bfbcfe817a84327f9373478767c3135b18159b547e0aafd903b" exitCode=0 Oct 30 18:33:50.234047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.233855 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" event={"ID":"91bacf95-f115-438a-a3cf-e7c753ec1a82","Type":"ContainerDied","Data":"424530e746905bfbcfe817a84327f9373478767c3135b18159b547e0aafd903b"} Oct 30 18:33:50.251623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.251592 4883 generic.go:356] "Generic (PLEG): container finished" podID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" containerID="ae60207c70c62d74f097143a236f5e58f5fd2f2b11a0196ed3493eb3fa656e99" exitCode=0 Oct 30 18:33:50.251830 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.251796 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" event={"ID":"d73d31fd-a3de-4549-8d6d-bcddd36d3a48","Type":"ContainerDied","Data":"ae60207c70c62d74f097143a236f5e58f5fd2f2b11a0196ed3493eb3fa656e99"} Oct 30 18:33:50.258863 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.257075 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:33:50.258863 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.258522 4883 generic.go:356] "Generic (PLEG): container finished" podID="48259124-69a9-4121-91cb-337d2724acb9" containerID="24f4d64d7f2f2025df2aeaeb7e5a7f5bf5eb2d3824429a40e56b78f73e1e8a16" exitCode=0 Oct 30 18:33:50.258863 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.258569 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" event={"ID":"48259124-69a9-4121-91cb-337d2724acb9","Type":"ContainerDied","Data":"24f4d64d7f2f2025df2aeaeb7e5a7f5bf5eb2d3824429a40e56b78f73e1e8a16"} Oct 30 18:33:50.303760 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.303728 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:33:50.314304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.314269 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9"] Oct 30 18:33:50.314863 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:33:50.314832 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7384b5_f1fc_40ea_9d0e_2ab3e19efdb6.slice/crio-4e3bb41b9d36d58786027dc13479e5969cc16db492f1a143ee384906570078ef WatchSource:0}: Error finding container 4e3bb41b9d36d58786027dc13479e5969cc16db492f1a143ee384906570078ef: Status 404 returned error can't find the container with id 4e3bb41b9d36d58786027dc13479e5969cc16db492f1a143ee384906570078ef Oct 30 18:33:50.321101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.321065 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:50.321215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.321138 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:50.321215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.321168 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:50.343122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343085 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48259124-69a9-4121-91cb-337d2724acb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48259124-69a9-4121-91cb-337d2724acb9-istio-podinfo\") pod \"48259124-69a9-4121-91cb-337d2724acb9\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " Oct 30 18:33:50.343305 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343204 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48259124-69a9-4121-91cb-337d2724acb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48259124-69a9-4121-91cb-337d2724acb9-istiod-ca-cert\") pod \"48259124-69a9-4121-91cb-337d2724acb9\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " Oct 30 18:33:50.343305 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343247 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-credential-socket\") pod \"91bacf95-f115-438a-a3cf-e7c753ec1a82\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " Oct 30 18:33:50.343425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343304 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-socket\") pod \"48259124-69a9-4121-91cb-337d2724acb9\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " Oct 30 18:33:50.343425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343366 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91bacf95-f115-438a-a3cf-e7c753ec1a82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bacf95-f115-438a-a3cf-e7c753ec1a82-istiod-ca-cert\") pod \"91bacf95-f115-438a-a3cf-e7c753ec1a82\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " Oct 30 18:33:50.343425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343417 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-kube-api-access-gvz79\" (UniqueName: \"kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-kube-api-access-gvz79\") pod \"48259124-69a9-4121-91cb-337d2724acb9\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " Oct 30 18:33:50.343588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343465 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-certs\") pod \"91bacf95-f115-438a-a3cf-e7c753ec1a82\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " Oct 30 18:33:50.343588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343516 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-kube-api-access-5qq52\" (UniqueName: \"kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-kube-api-access-5qq52\") pod \"91bacf95-f115-438a-a3cf-e7c753ec1a82\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " Oct 30 18:33:50.343699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343592 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-istio-token\" (UniqueName: \"kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-istio-token\") pod \"48259124-69a9-4121-91cb-337d2724acb9\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " Oct 30 18:33:50.343699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343646 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-podinfo\") pod \"91bacf95-f115-438a-a3cf-e7c753ec1a82\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " Oct 30 18:33:50.343810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343707 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-envoy\") pod \"48259124-69a9-4121-91cb-337d2724acb9\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " Oct 30 18:33:50.343810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343804 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-certs\") pod \"48259124-69a9-4121-91cb-337d2724acb9\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " Oct 30 18:33:50.343919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343848 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-socket\") pod \"91bacf95-f115-438a-a3cf-e7c753ec1a82\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " Oct 30 18:33:50.343919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343876 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-token\" (UniqueName: \"kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-token\") pod \"91bacf95-f115-438a-a3cf-e7c753ec1a82\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " Oct 30 18:33:50.344055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343922 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-data\") pod \"91bacf95-f115-438a-a3cf-e7c753ec1a82\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " Oct 30 18:33:50.344055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.343992 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-credential-socket\") pod \"48259124-69a9-4121-91cb-337d2724acb9\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " Oct 30 18:33:50.344055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.344047 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-data\") pod \"48259124-69a9-4121-91cb-337d2724acb9\" (UID: \"48259124-69a9-4121-91cb-337d2724acb9\") " Oct 30 18:33:50.344208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.344073 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-envoy\") pod \"91bacf95-f115-438a-a3cf-e7c753ec1a82\" (UID: \"91bacf95-f115-438a-a3cf-e7c753ec1a82\") " Oct 30 18:33:50.355293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.355207 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-socket" pod "91bacf95-f115-438a-a3cf-e7c753ec1a82" (UID: "91bacf95-f115-438a-a3cf-e7c753ec1a82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.360637 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.360609 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-socket" pod "48259124-69a9-4121-91cb-337d2724acb9" (UID: "48259124-69a9-4121-91cb-337d2724acb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.361218 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.361188 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-certs" pod "48259124-69a9-4121-91cb-337d2724acb9" (UID: "48259124-69a9-4121-91cb-337d2724acb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.361218 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.361198 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-data" pod "91bacf95-f115-438a-a3cf-e7c753ec1a82" (UID: "91bacf95-f115-438a-a3cf-e7c753ec1a82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.361408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.361334 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91bacf95-f115-438a-a3cf-e7c753ec1a82-istiod-ca-cert" pod "91bacf95-f115-438a-a3cf-e7c753ec1a82" (UID: "91bacf95-f115-438a-a3cf-e7c753ec1a82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:50.361551 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.361514 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48259124-69a9-4121-91cb-337d2724acb9-istiod-ca-cert" pod "48259124-69a9-4121-91cb-337d2724acb9" (UID: "48259124-69a9-4121-91cb-337d2724acb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:50.361664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.361640 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-credential-socket" pod "48259124-69a9-4121-91cb-337d2724acb9" (UID: "48259124-69a9-4121-91cb-337d2724acb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.361865 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.361848 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-data" pod "48259124-69a9-4121-91cb-337d2724acb9" (UID: "48259124-69a9-4121-91cb-337d2724acb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.362368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.362086 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-credential-socket" pod "91bacf95-f115-438a-a3cf-e7c753ec1a82" (UID: "91bacf95-f115-438a-a3cf-e7c753ec1a82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.363235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.363195 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-certs" pod "91bacf95-f115-438a-a3cf-e7c753ec1a82" (UID: "91bacf95-f115-438a-a3cf-e7c753ec1a82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.402002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.401949 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-kube-api-access-gvz79" pod "48259124-69a9-4121-91cb-337d2724acb9" (UID: "48259124-69a9-4121-91cb-337d2724acb9"). InnerVolumeSpecName "kube-api-access-gvz79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.402676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.402636 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-token" pod "91bacf95-f115-438a-a3cf-e7c753ec1a82" (UID: "91bacf95-f115-438a-a3cf-e7c753ec1a82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.433654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.405070 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-envoy" pod "48259124-69a9-4121-91cb-337d2724acb9" (UID: "48259124-69a9-4121-91cb-337d2724acb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.433654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.405135 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-istio-token" pod "48259124-69a9-4121-91cb-337d2724acb9" (UID: "48259124-69a9-4121-91cb-337d2724acb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.433654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.405249 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-podinfo" pod "91bacf95-f115-438a-a3cf-e7c753ec1a82" (UID: "91bacf95-f115-438a-a3cf-e7c753ec1a82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:50.433654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.405819 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-envoy" pod "91bacf95-f115-438a-a3cf-e7c753ec1a82" (UID: "91bacf95-f115-438a-a3cf-e7c753ec1a82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.433654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.405889 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48259124-69a9-4121-91cb-337d2724acb9-istio-podinfo" pod "48259124-69a9-4121-91cb-337d2724acb9" (UID: "48259124-69a9-4121-91cb-337d2724acb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:50.433654 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.424912 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-kube-api-access-5qq52" pod "91bacf95-f115-438a-a3cf-e7c753ec1a82" (UID: "91bacf95-f115-438a-a3cf-e7c753ec1a82"). InnerVolumeSpecName "kube-api-access-5qq52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.446234 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.446209 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvz79\" (UniqueName: \"kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-kube-api-access-gvz79\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.446359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.446350 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.446415 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.446407 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qq52\" (UniqueName: \"kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-kube-api-access-5qq52\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.446479 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.446472 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48259124-69a9-4121-91cb-337d2724acb9-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.446533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.446527 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.446591 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.446583 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.446645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.446638 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.446699 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.446690 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.446749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.446742 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.446800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.446794 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.446874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.446866 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.446973 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.446959 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.447051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.447024 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.447051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.447040 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48259124-69a9-4121-91cb-337d2724acb9-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.447161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.447053 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48259124-69a9-4121-91cb-337d2724acb9-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.447161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.447067 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bacf95-f115-438a-a3cf-e7c753ec1a82-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.447161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.447092 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48259124-69a9-4121-91cb-337d2724acb9-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.447314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.447168 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bacf95-f115-438a-a3cf-e7c753ec1a82-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.544133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.544097 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:33:50.546503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.546462 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:33:50.546625 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.546536 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" podUID="3ecdc930-9541-4a66-9967-4fb76ca616d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 18:33:50.548648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.548630 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-credential-socket\") pod \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " Oct 30 18:33:50.549535 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.549441 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-credential-socket" pod "d73d31fd-a3de-4549-8d6d-bcddd36d3a48" (UID: "d73d31fd-a3de-4549-8d6d-bcddd36d3a48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.549643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.549630 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-socket\") pod \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " Oct 30 18:33:50.549807 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.549785 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-socket" pod "d73d31fd-a3de-4549-8d6d-bcddd36d3a48" (UID: "d73d31fd-a3de-4549-8d6d-bcddd36d3a48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.549911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.549899 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-certs\") pod \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " Oct 30 18:33:50.550054 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.550038 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-certs" pod "d73d31fd-a3de-4549-8d6d-bcddd36d3a48" (UID: "d73d31fd-a3de-4549-8d6d-bcddd36d3a48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.550143 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.550132 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-envoy\") pod \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " Oct 30 18:33:50.550250 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.550241 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-data\") pod \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " Oct 30 18:33:50.550911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.550876 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-token\" (UniqueName: \"kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-token\") pod \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " Oct 30 18:33:50.551027 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.550945 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istiod-ca-cert\") pod \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " Oct 30 18:33:50.551090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.551036 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-kube-api-access-r6lzd\" (UniqueName: \"kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-kube-api-access-r6lzd\") pod \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " Oct 30 18:33:50.551146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.551095 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-podinfo\") pod \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\" (UID: \"d73d31fd-a3de-4549-8d6d-bcddd36d3a48\") " Oct 30 18:33:50.551835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.551806 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.551914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.551835 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.551914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.551850 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.552251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.552196 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-data" pod "d73d31fd-a3de-4549-8d6d-bcddd36d3a48" (UID: "d73d31fd-a3de-4549-8d6d-bcddd36d3a48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.552344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.552321 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istiod-ca-cert" pod "d73d31fd-a3de-4549-8d6d-bcddd36d3a48" (UID: "d73d31fd-a3de-4549-8d6d-bcddd36d3a48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:50.554696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.554673 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-token" pod "d73d31fd-a3de-4549-8d6d-bcddd36d3a48" (UID: "d73d31fd-a3de-4549-8d6d-bcddd36d3a48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.560312 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.560274 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-envoy" pod "d73d31fd-a3de-4549-8d6d-bcddd36d3a48" (UID: "d73d31fd-a3de-4549-8d6d-bcddd36d3a48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:50.605430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.598737 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-podinfo" pod "d73d31fd-a3de-4549-8d6d-bcddd36d3a48" (UID: "d73d31fd-a3de-4549-8d6d-bcddd36d3a48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:50.607600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.607566 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-kube-api-access-r6lzd" pod "d73d31fd-a3de-4549-8d6d-bcddd36d3a48" (UID: "d73d31fd-a3de-4549-8d6d-bcddd36d3a48"). InnerVolumeSpecName "kube-api-access-r6lzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:50.653283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.653162 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.653283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.653203 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.653283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.653217 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.653283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.653230 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.653283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.653243 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6lzd\" (UniqueName: \"kubernetes.io/projected/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-kube-api-access-r6lzd\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.653283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.653256 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73d31fd-a3de-4549-8d6d-bcddd36d3a48-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:33:50.712711 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.712671 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq"] Oct 30 18:33:50.715890 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.715844 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:50.716005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.715953 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:50.716086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:50.716001 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:51.288487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.288448 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" event={"ID":"3f40814a-3d5c-4e54-b5d8-8e9866d3b093","Type":"ContainerStarted","Data":"42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc"} Oct 30 18:33:51.289071 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.289044 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" event={"ID":"3f40814a-3d5c-4e54-b5d8-8e9866d3b093","Type":"ContainerStarted","Data":"a128a856318fceca6acb643e92d76345bb8c475df7e55067a49cd3b89c544887"} Oct 30 18:33:51.292413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.292388 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" event={"ID":"48259124-69a9-4121-91cb-337d2724acb9","Type":"ContainerDied","Data":"c17fd83a78cc7edcc738d40a42351b439c934211a3f1b422225ca456c938e81b"} Oct 30 18:33:51.292687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.292672 4883 scope.go:122] "RemoveContainer" containerID="24f4d64d7f2f2025df2aeaeb7e5a7f5bf5eb2d3824429a40e56b78f73e1e8a16" Oct 30 18:33:51.292919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.292613 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78" Oct 30 18:33:51.300194 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.300165 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" Oct 30 18:33:51.301213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.300178 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f" event={"ID":"91bacf95-f115-438a-a3cf-e7c753ec1a82","Type":"ContainerDied","Data":"dbb6ebd27ff13905a9209299cf00703fda1d16c253fd50a4656d581ba9cfd337"} Oct 30 18:33:51.303542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.303481 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" event={"ID":"d73d31fd-a3de-4549-8d6d-bcddd36d3a48","Type":"ContainerDied","Data":"da7ff5d34644e1314707a8b817cafc4045aa7b50167c03934041c8e1b9172d04"} Oct 30 18:33:51.303542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.303536 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw" Oct 30 18:33:51.307178 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.306899 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" event={"ID":"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6","Type":"ContainerStarted","Data":"dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7"} Oct 30 18:33:51.307178 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.306968 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" event={"ID":"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6","Type":"ContainerStarted","Data":"4e3bb41b9d36d58786027dc13479e5969cc16db492f1a143ee384906570078ef"} Oct 30 18:33:51.316527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.316501 4883 scope.go:122] "RemoveContainer" containerID="424530e746905bfbcfe817a84327f9373478767c3135b18159b547e0aafd903b" Oct 30 18:33:51.328410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.328390 4883 scope.go:122] "RemoveContainer" containerID="ae60207c70c62d74f097143a236f5e58f5fd2f2b11a0196ed3493eb3fa656e99" Oct 30 18:33:51.342022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.341067 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" podStartSLOduration=2.34104827 podStartE2EDuration="2.34104827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:51.338826115 +0000 UTC m=+2490.295829951" watchObservedRunningTime="2026-10-30 18:33:51.34104827 +0000 UTC m=+2490.298052107" Oct 30 18:33:51.403266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.403237 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f"] Oct 30 18:33:51.408721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.408694 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-hw24f"] Oct 30 18:33:51.441030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.440995 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw"] Oct 30 18:33:51.445224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.445189 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-wsfnw"] Oct 30 18:33:51.498609 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.498492 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" podStartSLOduration=2.498476079 podStartE2EDuration="2.498476079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:51.492668991 +0000 UTC m=+2490.449672828" watchObservedRunningTime="2026-10-30 18:33:51.498476079 +0000 UTC m=+2490.455479915" Oct 30 18:33:51.535282 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.535248 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78"] Oct 30 18:33:51.539224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.539197 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-77db82-openshift-default-5cc97b9779-sbn78"] Oct 30 18:33:51.546665 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.546639 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:33:51.546764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.546682 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" podUID="3ecdc930-9541-4a66-9967-4fb76ca616d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 18:33:51.848039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.847987 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:51.849157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.849021 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 18:33:51.849157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:51.849090 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" podUID="3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 18:33:52.162169 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:52.162085 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:52.163516 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:52.163487 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:52.163712 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:52.163687 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" podUID="3f40814a-3d5c-4e54-b5d8-8e9866d3b093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 18:33:52.546524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:52.546437 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:33:52.546524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:52.546503 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" podUID="3ecdc930-9541-4a66-9967-4fb76ca616d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 18:33:52.849954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:52.845110 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 18:33:52.849954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:52.845177 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" podUID="3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 18:33:53.162305 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:53.162228 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:53.162305 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:53.162278 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" podUID="3f40814a-3d5c-4e54-b5d8-8e9866d3b093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 18:33:53.217960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:53.217309 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48259124-69a9-4121-91cb-337d2724acb9" path="/var/lib/kubelet/pods/48259124-69a9-4121-91cb-337d2724acb9/volumes" Oct 30 18:33:53.218169 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:53.218105 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91bacf95-f115-438a-a3cf-e7c753ec1a82" path="/var/lib/kubelet/pods/91bacf95-f115-438a-a3cf-e7c753ec1a82/volumes" Oct 30 18:33:53.222067 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:53.219296 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" path="/var/lib/kubelet/pods/d73d31fd-a3de-4549-8d6d-bcddd36d3a48/volumes" Oct 30 18:33:53.551728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:53.551652 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:53.552114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:53.552080 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:53.552659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:53.552642 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:33:53.845485 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:53.844970 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 18:33:53.845485 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:53.845025 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" podUID="3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 18:33:54.162206 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:54.162123 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:54.162206 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:54.162173 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" podUID="3f40814a-3d5c-4e54-b5d8-8e9866d3b093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 18:33:54.844058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:54.844027 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 18:33:54.844522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:54.844072 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" podUID="3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 18:33:55.162548 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:55.162458 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:55.162548 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:55.162515 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" podUID="3f40814a-3d5c-4e54-b5d8-8e9866d3b093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 18:33:55.844613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:55.844570 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 18:33:55.845150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:55.844640 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" podUID="3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 18:33:56.162751 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:56.162658 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:56.162751 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:56.162725 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" podUID="3f40814a-3d5c-4e54-b5d8-8e9866d3b093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 18:33:56.844405 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:56.844368 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 18:33:56.844701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:56.844673 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" podUID="3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 18:33:57.036162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:57.036127 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w"] Oct 30 18:33:57.047144 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:57.047110 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04839d99-vmn5w"] Oct 30 18:33:57.162656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:57.162560 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 18:33:57.162656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:57.162624 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" podUID="3f40814a-3d5c-4e54-b5d8-8e9866d3b093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 18:33:57.229740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:57.229704 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5201f205-e45c-4fc6-955d-8a65329dcfca" path="/var/lib/kubelet/pods/5201f205-e45c-4fc6-955d-8a65329dcfca/volumes" Oct 30 18:33:57.941808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:57.941772 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:57.943029 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:57.943010 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:57.944119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:57.944098 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:33:58.239947 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:58.239846 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:58.240806 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:58.240781 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:33:58.241740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:33:58.241711 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:34:03.169279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:03.169243 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr"] Oct 30 18:34:03.181630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:03.181602 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-602e18-mhckr"] Oct 30 18:34:05.083231 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:34:05.083195 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:34:05.179540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:05.179501 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3" path="/var/lib/kubelet/pods/c80d2586-bf95-4d83-8ed1-8f7fd9eabcc3/volumes" Oct 30 18:34:10.039953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:10.039907 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb"] Oct 30 18:34:10.052970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:10.052926 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bcb652-97tnb"] Oct 30 18:34:11.179556 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:11.179525 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fea8dd3-63ff-4471-8e0c-53bc6b0b803f" path="/var/lib/kubelet/pods/4fea8dd3-63ff-4471-8e0c-53bc6b0b803f/volumes" Oct 30 18:34:26.959580 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.959546 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87"] Oct 30 18:34:26.960149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.960135 4883 cpu_manager.go:414] "containerMap: removing container" podUID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" containerName="istio-proxy" Oct 30 18:34:26.960226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.960150 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" containerName="istio-proxy" Oct 30 18:34:26.960226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.960177 4883 cpu_manager.go:414] "containerMap: removing container" podUID="48259124-69a9-4121-91cb-337d2724acb9" containerName="istio-proxy" Oct 30 18:34:26.960226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.960183 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48259124-69a9-4121-91cb-337d2724acb9" containerName="istio-proxy" Oct 30 18:34:26.960226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.960210 4883 cpu_manager.go:414] "containerMap: removing container" podUID="91bacf95-f115-438a-a3cf-e7c753ec1a82" containerName="istio-proxy" Oct 30 18:34:26.960226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.960216 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91bacf95-f115-438a-a3cf-e7c753ec1a82" containerName="istio-proxy" Oct 30 18:34:26.960412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.960338 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="91bacf95-f115-438a-a3cf-e7c753ec1a82" containerName="istio-proxy" Oct 30 18:34:26.960412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.960350 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73d31fd-a3de-4549-8d6d-bcddd36d3a48" containerName="istio-proxy" Oct 30 18:34:26.960412 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.960362 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="48259124-69a9-4121-91cb-337d2724acb9" containerName="istio-proxy" Oct 30 18:34:26.963235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.963215 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:26.997717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.997687 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87"] Oct 30 18:34:26.999702 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.999672 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c816bfc8-985e-4cf5-865c-5adee66fb87d-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:26.999839 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.999727 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:26.999839 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.999819 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:26.999966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:26.999871 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.000033 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.000016 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.000089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.000064 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.000160 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.000145 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwqrm\" (UniqueName: \"kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-kube-api-access-vwqrm\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.000213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.000178 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.000296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.000271 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.102301 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.102263 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.102489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.102326 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.102489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.102382 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwqrm\" (UniqueName: \"kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-kube-api-access-vwqrm\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.102489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.102411 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.102489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.102458 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.102709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.102614 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c816bfc8-985e-4cf5-865c-5adee66fb87d-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.102709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.102656 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.102709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.102692 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.102709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.102696 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.102911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.102789 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.102976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.102958 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.103101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.103077 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.103286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.103148 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.103392 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.103326 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c816bfc8-985e-4cf5-865c-5adee66fb87d-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.105408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.105385 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.105506 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.105386 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.128792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.128757 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwqrm\" (UniqueName: \"kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-kube-api-access-vwqrm\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.129440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.129415 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.277911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.277821 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:27.601222 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.601191 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87"] Oct 30 18:34:27.601995 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:34:27.601968 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc816bfc8_985e_4cf5_865c_5adee66fb87d.slice/crio-f1f2f4048cd1e27859b51dafb5873cbdc546b97d729926426b1f84a6d724cf62 WatchSource:0}: Error finding container f1f2f4048cd1e27859b51dafb5873cbdc546b97d729926426b1f84a6d724cf62: Status 404 returned error can't find the container with id f1f2f4048cd1e27859b51dafb5873cbdc546b97d729926426b1f84a6d724cf62 Oct 30 18:34:27.605003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.604970 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:34:27.605105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.605038 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:34:27.605105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.605064 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:34:27.730208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.730167 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" event={"ID":"c816bfc8-985e-4cf5-865c-5adee66fb87d","Type":"ContainerStarted","Data":"77895149bea2ab91195cffdcb7fc8e824a3e2c67a7ea5d137350255d51baece4"} Oct 30 18:34:27.730415 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.730236 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" event={"ID":"c816bfc8-985e-4cf5-865c-5adee66fb87d","Type":"ContainerStarted","Data":"f1f2f4048cd1e27859b51dafb5873cbdc546b97d729926426b1f84a6d724cf62"} Oct 30 18:34:27.782232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:27.782175 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" podStartSLOduration=1.782157158 podStartE2EDuration="1.782157158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:27.778420735 +0000 UTC m=+2526.735424571" watchObservedRunningTime="2026-10-30 18:34:27.782157158 +0000 UTC m=+2526.739161061" Oct 30 18:34:28.284203 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:28.284113 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:28.289390 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:28.289047 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:34:28.289390 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:28.289121 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" podUID="c816bfc8-985e-4cf5-865c-5adee66fb87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 18:34:29.284963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:29.282310 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:34:29.284963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:29.282375 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" podUID="c816bfc8-985e-4cf5-865c-5adee66fb87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 18:34:30.281214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:30.281174 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:34:30.281512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:30.281475 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" podUID="c816bfc8-985e-4cf5-865c-5adee66fb87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 18:34:31.289121 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:31.289081 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:34:31.289541 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:31.289134 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" podUID="c816bfc8-985e-4cf5-865c-5adee66fb87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 18:34:32.295378 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:32.295338 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:34:32.296199 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:32.296160 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" podUID="c816bfc8-985e-4cf5-865c-5adee66fb87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 18:34:32.831769 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:32.831734 4883 scope.go:122] "RemoveContainer" containerID="9c16dd9d0fbd6ff878fe109598c7310884142171de417609c9970034c8d97c47" Oct 30 18:34:32.842999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:32.842975 4883 scope.go:122] "RemoveContainer" containerID="54f77417a44fc6263ca37ffa7715f3c1e0d387b5a1a02fa9813987a4bf2de54d" Oct 30 18:34:32.853040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:32.853016 4883 scope.go:122] "RemoveContainer" containerID="44b15cc160dfaf8b764493ef9c2a5199f8292d04497ca1e0c9274200c7838310" Oct 30 18:34:32.889452 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:32.884110 4883 scope.go:122] "RemoveContainer" containerID="246ce382117226ba8eace738b572b7a64b9c8b9d25850279ae787778f7dc96fe" Oct 30 18:34:32.900970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:32.900944 4883 scope.go:122] "RemoveContainer" containerID="536d4c70252da8ce86910ca74ac60c3d62fad5f77f146c3d190af8c503995f3a" Oct 30 18:34:32.910467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:32.910445 4883 scope.go:122] "RemoveContainer" containerID="070820b104a0d0d45f99538decb8e9a0da1dcd1a06a5864759da80b0cb7c2d33" Oct 30 18:34:32.919665 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:32.919642 4883 scope.go:122] "RemoveContainer" containerID="ca6c1f71917204c39e18a90db63c39db0cb067d3b12cd8a6bb3df08fcd740e5a" Oct 30 18:34:32.929134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:32.929112 4883 scope.go:122] "RemoveContainer" containerID="1587bbb9849e73baa967e8e49181818b0f75dfa64d99b453982c3980715937b1" Oct 30 18:34:33.289175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:33.289098 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:34:33.289377 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:33.289175 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" podUID="c816bfc8-985e-4cf5-865c-5adee66fb87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 18:34:34.296929 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:34.296882 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:34.297449 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:34.297354 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:34.298382 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:34.298359 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:34:35.079455 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:34:35.079417 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:34:52.329762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:34:52.329727 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg"] Oct 30 18:35:05.079388 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:35:05.079305 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:35:07.363815 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.363771 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" containerID="cri-o://b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17" gracePeriod=45 Oct 30 18:35:07.660867 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.660838 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:35:07.739367 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.739333 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tmp-dir\") pod \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " Oct 30 18:35:07.739572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.739540 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-model-cache\") pod \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " Oct 30 18:35:07.739641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.739602 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tmp-dir" pod "ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" (UID: "ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:07.739709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.739651 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-dshm\") pod \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " Oct 30 18:35:07.739766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.739717 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tls-certs\") pod \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " Oct 30 18:35:07.739823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.739791 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-home\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-home\") pod \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " Oct 30 18:35:07.739823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.739802 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-model-cache" pod "ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" (UID: "ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:07.739957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.739832 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-kube-api-access-bnwkq\" (UniqueName: \"kubernetes.io/projected/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-kube-api-access-bnwkq\") pod \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\" (UID: \"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7\") " Oct 30 18:35:07.740056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.740025 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-home" pod "ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" (UID: "ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:07.740630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.740604 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:07.740765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.740633 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:07.740765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.740649 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:07.742102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.742078 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-dshm" pod "ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" (UID: "ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:07.742361 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.742341 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tls-certs" pod "ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" (UID: "ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:35:07.742361 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.742354 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-kube-api-access-bnwkq" pod "ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" (UID: "ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7"). InnerVolumeSpecName "kube-api-access-bnwkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:07.842226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.842188 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:07.842226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.842219 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:07.842226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:07.842229 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnwkq\" (UniqueName: \"kubernetes.io/projected/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7-kube-api-access-bnwkq\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:08.109922 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:08.109887 4883 generic.go:356] "Generic (PLEG): container finished" podID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerID="b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17" exitCode=0 Oct 30 18:35:08.110115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:08.109972 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" event={"ID":"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7","Type":"ContainerDied","Data":"b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17"} Oct 30 18:35:08.110115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:08.110020 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" event={"ID":"ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7","Type":"ContainerDied","Data":"9874896420e54aca35838aa76f34c9eea13b231dfaf96ef0fa6bd423ad432f67"} Oct 30 18:35:08.110115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:08.110038 4883 scope.go:122] "RemoveContainer" containerID="b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17" Oct 30 18:35:08.110115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:08.109984 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg" Oct 30 18:35:08.121495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:08.121475 4883 scope.go:122] "RemoveContainer" containerID="b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17" Oct 30 18:35:08.121759 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:35:08.121739 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17\": container with ID starting with b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17 not found: ID does not exist" containerID="b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17" Oct 30 18:35:08.121830 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:08.121771 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17"} err="failed to get container status \"b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17\": rpc error: code = NotFound desc = could not find container \"b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17\": container with ID starting with b95d720ab705c994eacc0ccf25a1622006596eca9494753f5fbdf53388c10b17 not found: ID does not exist" Oct 30 18:35:08.166002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:08.165947 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg"] Oct 30 18:35:08.178783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:08.178754 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ac632e/auth-test-model-ac632e-kserve-555fd95b85-kjqwg"] Oct 30 18:35:09.180040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:09.180001 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" path="/var/lib/kubelet/pods/ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7/volumes" Oct 30 18:35:09.432026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:09.431946 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d"] Oct 30 18:35:24.454503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.454465 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" containerID="cri-o://1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff" gracePeriod=45 Oct 30 18:35:24.796677 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.796642 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:35:24.855476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.855442 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-kube-api-access-988b7\" (UniqueName: \"kubernetes.io/projected/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-kube-api-access-988b7\") pod \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " Oct 30 18:35:24.855659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.855513 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-dshm\") pod \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " Oct 30 18:35:24.855659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.855575 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-model-cache\") pod \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " Oct 30 18:35:24.855659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.855644 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tmp-dir\") pod \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " Oct 30 18:35:24.855849 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.855687 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-home\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-home\") pod \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " Oct 30 18:35:24.855849 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.855804 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tls-certs\") pod \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\" (UID: \"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96\") " Oct 30 18:35:24.856037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.856003 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-model-cache" pod "3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" (UID: "3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:24.856226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.856204 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tmp-dir" pod "3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" (UID: "3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:24.856716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.856694 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:24.856829 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.856743 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:24.857038 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.857016 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-home" pod "3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" (UID: "3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:24.858262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.858234 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-dshm" pod "3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" (UID: "3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:24.859224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.859201 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tls-certs" pod "3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" (UID: "3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:35:24.859318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.859267 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-kube-api-access-988b7" pod "3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" (UID: "3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96"). InnerVolumeSpecName "kube-api-access-988b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:24.957857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.957818 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-988b7\" (UniqueName: \"kubernetes.io/projected/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-kube-api-access-988b7\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:24.957857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.957852 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:24.957857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.957862 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:24.958107 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:24.957872 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:25.277589 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:25.277558 4883 generic.go:356] "Generic (PLEG): container finished" podID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerID="1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff" exitCode=0 Oct 30 18:35:25.278010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:25.277617 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" Oct 30 18:35:25.278010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:25.277616 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" event={"ID":"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96","Type":"ContainerDied","Data":"1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff"} Oct 30 18:35:25.278010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:25.277667 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d" event={"ID":"3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96","Type":"ContainerDied","Data":"d58be5277a448f38651c41c4ca5ab6d5aa46445ffc12adc4f4e18e744f47ec48"} Oct 30 18:35:25.278010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:25.277684 4883 scope.go:122] "RemoveContainer" containerID="1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff" Oct 30 18:35:25.292401 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:25.290825 4883 scope.go:122] "RemoveContainer" containerID="1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff" Oct 30 18:35:25.292791 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:35:25.292766 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff\": container with ID starting with 1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff not found: ID does not exist" containerID="1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff" Oct 30 18:35:25.292882 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:25.292797 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff"} err="failed to get container status \"1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff\": rpc error: code = NotFound desc = could not find container \"1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff\": container with ID starting with 1ea1ae5da5e09ef861b0ca9e993c3d44ddc3497978603b5e31afde18ae4d61ff not found: ID does not exist" Oct 30 18:35:25.336006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:25.335975 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d"] Oct 30 18:35:25.349641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:25.349614 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5cb3d5/auth-test-model-5cb3d5-kserve-fcbf9c497-mv48d"] Oct 30 18:35:27.178780 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:27.178750 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" path="/var/lib/kubelet/pods/3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96/volumes" Oct 30 18:35:33.033200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:33.033160 4883 scope.go:122] "RemoveContainer" containerID="e0a90f5772055ff3c8ec3c229fbb2ce2a1115ee55495c9f28327df219644870a" Oct 30 18:35:33.043281 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:33.043259 4883 scope.go:122] "RemoveContainer" containerID="470d1dd7d61fcbff7068d0cdd69cb2f374a82b0b638ccdaa2096b00b015c47ba" Oct 30 18:35:33.053192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:33.053165 4883 scope.go:122] "RemoveContainer" containerID="0a43171a0de352061d451fdb98be646803aca5fd365f3504eda3b1442437861c" Oct 30 18:35:33.062728 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:33.062706 4883 scope.go:122] "RemoveContainer" containerID="f08c4ba2fa8908df9aaf58257c378d5e2190e4fdf6d4dfa8b2572d3241b64b93" Oct 30 18:35:35.318960 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:35:35.318108 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:35:35.507959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:35.500059 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:35.532959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:35.530090 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:35:36.384126 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:36.383025 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:36.417182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:36.417130 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": context deadline exceeded" Oct 30 18:35:36.512956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:36.512043 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:36.512956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:36.512677 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:35:36.537996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:36.533134 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:36.537996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:36.534515 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:35:36.919958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:36.919645 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:36.937960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:36.925017 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:36.937960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:36.927305 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:35:36.937960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:36.928776 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": context deadline exceeded" Oct 30 18:35:37.059959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:36.986611 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.059959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:37.059644 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:35:37.092293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:37.092235 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.218027 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:37.194449 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:35:37.219230 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:37.219185 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.444976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:37.409067 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.447267 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:37.427335 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.523015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:37.519269 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:35:37.523015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:37.519328 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:35:37.523352 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:37.523318 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:37.534459 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:37.524342 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:35:37.572430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:37.541303 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.091988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.057416 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:35:38.091988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.058160 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:35:38.091988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.059848 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.091988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.060374 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:35:38.091988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.089177 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": context deadline exceeded" Oct 30 18:35:38.109152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.093401 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:35:38.110162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.109453 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:35:38.111676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.111651 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:35:38.259085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.111882 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:35:38.259085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.176371 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.259085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.185240 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.261382 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.261333 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.334952 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.329107 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:35:38.359233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.359114 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:35:38.385198 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.385168 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:35:38.385543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.385524 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:35:38.385766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.385751 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:35:38.415545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.362474 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Oct 30 18:35:38.469986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.446050 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.469986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.446467 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded" Oct 30 18:35:38.557008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.506534 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:38.844204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.821115 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.844204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.569970 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:35:38.844204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.558192 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.844204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.573491 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:38.844204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.573752 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:35:38.844204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:38.822708 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:35:39.111017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.069070 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.111017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.069477 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.111017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.092067 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.111017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.094173 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.111017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.094577 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:35:39.158958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.130125 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.350236 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.350196 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.350988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.350956 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:35:39.353785 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.353758 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:35:39.366471 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.366363 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:35:39.402353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.402300 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:35:39.441959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.421029 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.441959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.421253 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.502435 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.501035 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.513958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.503189 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Oct 30 18:35:39.667048 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.631454 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.804139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.804091 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.823150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.823096 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:39.823766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.823730 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:35:39.824200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.824177 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": context deadline exceeded" Oct 30 18:35:39.845156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.825010 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:35:40.004026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:39.999023 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.139404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.139356 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:35:40.166000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.139967 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:35:40.166583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.147212 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.186782 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.149101 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": context deadline exceeded" Oct 30 18:35:40.241974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.149255 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:35:40.242522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.242494 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:35:40.359960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.356305 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:35:40.510549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.408355 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.511198 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.511157 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.560456 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.536106 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.561382 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.546169 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.587339 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.560742 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 30 18:35:40.847017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.826259 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:35:40.847017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.827206 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:35:40.847017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.828002 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:40.847017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:40.838041 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.143197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.143078 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.169127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.169065 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.173122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.173075 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.192555 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.192504 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.195100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.195057 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:41.195308 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.195283 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.247960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.243798 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.247960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.244167 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:35:41.247960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.244581 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:35:41.256217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.256181 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:41.256462 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.256435 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.361326 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.361282 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.416172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.416065 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.512852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.512793 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.590112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.590062 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:35:41.829231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.828143 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:35:41.829231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.828701 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.833087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.829998 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:41.840041 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:41.840002 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:35:42.147118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.147070 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.173107 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.173064 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.184545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.184503 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": context deadline exceeded" Oct 30 18:35:42.198661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.198604 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.246158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.246101 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:35:42.246888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.246388 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.248655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.247418 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:35:42.364237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.362749 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:35:42.440019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.439881 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:35:42.577099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.577042 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.625601 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.601509 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:35:42.863966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.862655 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.878049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.878005 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:42.878657 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.878627 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:35:42.879900 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:42.879875 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": context deadline exceeded" Oct 30 18:35:43.271150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.271041 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:35:43.274972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.274919 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.298956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.276187 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:35:43.318328 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.295198 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.319018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.295255 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": context deadline exceeded" Oct 30 18:35:43.319388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.295276 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 18:35:43.369395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.369353 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.464958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.446011 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:43.464958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.446067 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.489955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.466838 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:35:43.653201 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.653156 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.654077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.653576 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.928576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.896033 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:43.928576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.896418 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:35:43.928576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.896690 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:35:44.039967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:43.904750 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:35:44.085158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.066377 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:35:44.325112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.325007 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.351956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.349541 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.351956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.350032 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:35:44.351956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.350375 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Oct 30 18:35:44.400028 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.399979 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.426226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.426110 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.508958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.489347 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:35:44.511317 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.511280 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:35:44.681958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.671476 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.681958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.673443 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:44.771750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.771715 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:35:44.817773 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.814515 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:35:44.917951 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.900791 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": context deadline exceeded" Oct 30 18:35:44.928953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.926346 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 18:35:44.968811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:44.968727 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:35:45.006952 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:45.002051 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:35:45.040197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:45.040168 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:35:45.054955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:45.051236 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:45.280347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:45.280264 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:35:45.280634 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:45.280616 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:35:45.280902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:45.280881 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:35:45.281093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:45.281079 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:35:45.281231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:45.281220 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:35:45.281346 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:45.281335 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:35:45.281463 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:45.281452 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:35:48.958549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.958511 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx"] Oct 30 18:35:48.959413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.959389 4883 cpu_manager.go:414] "containerMap: removing container" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" Oct 30 18:35:48.959413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.959411 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" Oct 30 18:35:48.959594 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.959469 4883 cpu_manager.go:414] "containerMap: removing container" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" Oct 30 18:35:48.959594 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.959475 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" Oct 30 18:35:48.959594 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.959577 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5d200e-7bb3-4a5e-9890-29eeeaa67ef7" containerName="main" Oct 30 18:35:48.959594 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.959587 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7f3b3e-d9a9-4c88-a79e-06fba5eb9d96" containerName="main" Oct 30 18:35:48.963498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.963469 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:48.986617 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.986589 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:35:48.986824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.986588 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:35:48.986824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.986689 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:35:48.992790 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.992769 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:35:48.999030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:48.999004 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx"] Oct 30 18:35:49.029158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:49.029124 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx\" (UID: \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:49.029365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:49.029293 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx\" (UID: \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:49.029503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:49.029479 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99mg\" (UniqueName: \"kubernetes.io/projected/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-kube-api-access-z99mg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx\" (UID: \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:49.131599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:49.131558 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx\" (UID: \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:49.132008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:49.131983 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99mg\" (UniqueName: \"kubernetes.io/projected/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-kube-api-access-z99mg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx\" (UID: \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:49.134021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:49.133996 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx\" (UID: \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:49.135112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:49.135090 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx\" (UID: \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:49.144390 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:49.144359 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx\" (UID: \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:49.161044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:49.161014 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99mg\" (UniqueName: \"kubernetes.io/projected/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-kube-api-access-z99mg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx\" (UID: \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:49.282817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:49.282725 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:49.618113 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:35:49.618080 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0e6ecd_99b6_418b_a671_fa7298dd30f8.slice/crio-e13146155adfd85073027e71c9725323f43691d5007b000bafa1cca7335340ae WatchSource:0}: Error finding container e13146155adfd85073027e71c9725323f43691d5007b000bafa1cca7335340ae: Status 404 returned error can't find the container with id e13146155adfd85073027e71c9725323f43691d5007b000bafa1cca7335340ae Oct 30 18:35:49.618718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:49.618692 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx"] Oct 30 18:35:50.319450 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:50.319357 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" event={"ID":"0f0e6ecd-99b6-418b-a671-fa7298dd30f8","Type":"ContainerStarted","Data":"40518a3a1321c0d4d0ce479b1473ec942f6dcc95a4698431b955c20ec0362c65"} Oct 30 18:35:50.319450 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:50.319418 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" event={"ID":"0f0e6ecd-99b6-418b-a671-fa7298dd30f8","Type":"ContainerStarted","Data":"e13146155adfd85073027e71c9725323f43691d5007b000bafa1cca7335340ae"} Oct 30 18:35:50.366138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:50.366079 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" podStartSLOduration=2.366064744 podStartE2EDuration="2.366064744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:50.362128278 +0000 UTC m=+2609.319132104" watchObservedRunningTime="2026-10-30 18:35:50.366064744 +0000 UTC m=+2609.323068578" Oct 30 18:35:51.328331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:51.328297 4883 generic.go:356] "Generic (PLEG): container finished" podID="0f0e6ecd-99b6-418b-a671-fa7298dd30f8" containerID="40518a3a1321c0d4d0ce479b1473ec942f6dcc95a4698431b955c20ec0362c65" exitCode=0 Oct 30 18:35:51.328722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:51.328388 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" event={"ID":"0f0e6ecd-99b6-418b-a671-fa7298dd30f8","Type":"ContainerDied","Data":"40518a3a1321c0d4d0ce479b1473ec942f6dcc95a4698431b955c20ec0362c65"} Oct 30 18:35:52.501536 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:52.501509 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:52.592567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:52.592538 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-openshift-ca\") pod \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\" (UID: \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\") " Oct 30 18:35:52.592735 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:52.592669 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-xks-ca\") pod \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\" (UID: \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\") " Oct 30 18:35:52.592826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:52.592800 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-kube-api-access-z99mg\" (UniqueName: \"kubernetes.io/projected/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-kube-api-access-z99mg\") pod \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\" (UID: \"0f0e6ecd-99b6-418b-a671-fa7298dd30f8\") " Oct 30 18:35:52.592920 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:52.592896 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-openshift-ca" pod "0f0e6ecd-99b6-418b-a671-fa7298dd30f8" (UID: "0f0e6ecd-99b6-418b-a671-fa7298dd30f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:35:52.593658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:52.593621 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:52.594828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:52.594798 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-xks-ca" pod "0f0e6ecd-99b6-418b-a671-fa7298dd30f8" (UID: "0f0e6ecd-99b6-418b-a671-fa7298dd30f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:35:52.595172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:52.595149 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-kube-api-access-z99mg" pod "0f0e6ecd-99b6-418b-a671-fa7298dd30f8" (UID: "0f0e6ecd-99b6-418b-a671-fa7298dd30f8"). InnerVolumeSpecName "kube-api-access-z99mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:52.695406 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:52.695374 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:52.695406 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:52.695401 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z99mg\" (UniqueName: \"kubernetes.io/projected/0f0e6ecd-99b6-418b-a671-fa7298dd30f8-kube-api-access-z99mg\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:35:53.423852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:53.423812 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" event={"ID":"0f0e6ecd-99b6-418b-a671-fa7298dd30f8","Type":"ContainerDied","Data":"e13146155adfd85073027e71c9725323f43691d5007b000bafa1cca7335340ae"} Oct 30 18:35:53.424075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:53.423901 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e13146155adfd85073027e71c9725323f43691d5007b000bafa1cca7335340ae" Oct 30 18:35:53.424075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:53.424001 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx" Oct 30 18:35:57.312916 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.312875 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf"] Oct 30 18:35:57.313738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.313715 4883 cpu_manager.go:414] "containerMap: removing container" podUID="0f0e6ecd-99b6-418b-a671-fa7298dd30f8" containerName="revoke-keys" Oct 30 18:35:57.313738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.313738 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0e6ecd-99b6-418b-a671-fa7298dd30f8" containerName="revoke-keys" Oct 30 18:35:57.313907 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.313898 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0e6ecd-99b6-418b-a671-fa7298dd30f8" containerName="revoke-keys" Oct 30 18:35:57.317567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.317546 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.365467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.365424 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf"] Oct 30 18:35:57.381230 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.381193 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.381378 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.381249 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.381378 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.381340 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.381515 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.381498 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.381598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.381583 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxj4\" (UniqueName: \"kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-kube-api-access-7rxj4\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.381676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.381662 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.381727 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.381710 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.381808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.381782 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.381846 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.381833 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.483276 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.483245 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.483445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.483310 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxj4\" (UniqueName: \"kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-kube-api-access-7rxj4\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.483445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.483383 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.483445 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.483423 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.483578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.483476 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.483578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.483509 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.483666 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.483596 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.483666 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.483622 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.483780 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.483660 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-data\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.483780 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.483702 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.483780 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.483716 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-credential-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.483980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.483803 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-certs\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.484317 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.484298 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-socket\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.484418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.484299 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istiod-ca-cert\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.485809 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.485788 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-envoy\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.486052 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.486034 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-podinfo\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.507790 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.507767 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-token\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.507948 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.507851 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxj4\" (UniqueName: \"kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-kube-api-access-7rxj4\") pod \"e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.632338 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.632307 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:57.955190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.955160 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf"] Oct 30 18:35:57.956461 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:35:57.956437 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081fffe0_7e1f_4562_b4c8_d8cb29c045bb.slice/crio-59189e1d6a193fbb05af7d9e58948499882736f4c2b6522b1e436e99160da8e7 WatchSource:0}: Error finding container 59189e1d6a193fbb05af7d9e58948499882736f4c2b6522b1e436e99160da8e7: Status 404 returned error can't find the container with id 59189e1d6a193fbb05af7d9e58948499882736f4c2b6522b1e436e99160da8e7 Oct 30 18:35:57.959142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.959104 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:35:57.959240 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.959173 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:35:57.959240 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:57.959209 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:35:58.461550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:58.461235 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" event={"ID":"081fffe0-7e1f-4562-b4c8-d8cb29c045bb","Type":"ContainerStarted","Data":"80df7a1577536df94586f4a26103032a034811ff425e5e7f0edb3db9b99a994e"} Oct 30 18:35:58.461550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:58.461299 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" event={"ID":"081fffe0-7e1f-4562-b4c8-d8cb29c045bb","Type":"ContainerStarted","Data":"59189e1d6a193fbb05af7d9e58948499882736f4c2b6522b1e436e99160da8e7"} Oct 30 18:35:58.526839 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:58.526773 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" podStartSLOduration=1.526753448 podStartE2EDuration="1.526753448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:58.525221508 +0000 UTC m=+2617.482225349" watchObservedRunningTime="2026-10-30 18:35:58.526753448 +0000 UTC m=+2617.483757283" Oct 30 18:35:58.633171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:58.633133 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:35:59.634411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:59.634374 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:35:59.634972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:35:59.634433 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" podUID="081fffe0-7e1f-4562-b4c8-d8cb29c045bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 18:36:00.643953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:00.642510 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:36:00.643953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:00.642581 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" podUID="081fffe0-7e1f-4562-b4c8-d8cb29c045bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 18:36:01.633324 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:01.633271 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:36:01.633549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:01.633352 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" podUID="081fffe0-7e1f-4562-b4c8-d8cb29c045bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 18:36:01.674642 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:01.674532 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx"] Oct 30 18:36:01.681286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:01.681253 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ac632e-bfflx"] Oct 30 18:36:02.632683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:02.632646 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:36:02.632848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:02.632708 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" podUID="081fffe0-7e1f-4562-b4c8-d8cb29c045bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 18:36:03.230958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:03.230587 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f0e6ecd-99b6-418b-a671-fa7298dd30f8" path="/var/lib/kubelet/pods/0f0e6ecd-99b6-418b-a671-fa7298dd30f8/volumes" Oct 30 18:36:03.633187 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:03.633154 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:36:03.633339 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:03.633216 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" podUID="081fffe0-7e1f-4562-b4c8-d8cb29c045bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 18:36:04.637802 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:04.637770 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:36:04.639277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:04.639253 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:36:04.640209 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:04.640188 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:36:05.177151 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:36:05.177086 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:36:06.426349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.426316 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk"] Oct 30 18:36:06.432775 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.432752 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.456096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.456066 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk"] Oct 30 18:36:06.524782 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.524749 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.524964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.524875 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.524964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.524918 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.525047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.524959 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.525047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.525022 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868aa718-8104-4403-86dd-59b3c3940966-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.525114 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.525051 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868aa718-8104-4403-86dd-59b3c3940966-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.525149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.525131 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.525208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.525194 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.525343 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.525330 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86b8h\" (UniqueName: \"kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-kube-api-access-86b8h\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.626614 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.626582 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86b8h\" (UniqueName: \"kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-kube-api-access-86b8h\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.626816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.626633 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.627081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.627046 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.627081 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.627079 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.627279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.627110 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.627279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.627139 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.627279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.627167 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868aa718-8104-4403-86dd-59b3c3940966-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.627279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.627191 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868aa718-8104-4403-86dd-59b3c3940966-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.627500 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.627279 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-certs\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.627500 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.627309 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.627500 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.627360 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.627500 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.627461 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-credential-socket\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.627999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.627971 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-data\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.628105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.628096 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868aa718-8104-4403-86dd-59b3c3940966-istiod-ca-cert\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.630118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.630088 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-envoy\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.630243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.630223 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868aa718-8104-4403-86dd-59b3c3940966-istio-podinfo\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.651374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.651345 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-istio-token\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.651489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.651387 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86b8h\" (UniqueName: \"kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-kube-api-access-86b8h\") pod \"e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:06.750993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:06.750828 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:07.102165 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:07.102135 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk"] Oct 30 18:36:07.103421 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:36:07.103393 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868aa718_8104_4403_86dd_59b3c3940966.slice/crio-ac21068cef3ba061dcd476e6604b7ef87d032e70339bfb7dc7f8bff88ed6b3ef WatchSource:0}: Error finding container ac21068cef3ba061dcd476e6604b7ef87d032e70339bfb7dc7f8bff88ed6b3ef: Status 404 returned error can't find the container with id ac21068cef3ba061dcd476e6604b7ef87d032e70339bfb7dc7f8bff88ed6b3ef Oct 30 18:36:07.106924 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:07.106892 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:36:07.107010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:07.106972 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:36:07.107010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:07.107005 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:36:07.643006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:07.642967 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" event={"ID":"868aa718-8104-4403-86dd-59b3c3940966","Type":"ContainerStarted","Data":"0ce268a1d2d3533bd456a6f52d27aa2437095ef171e6ed4446c830f39e874a81"} Oct 30 18:36:07.643497 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:07.643037 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" event={"ID":"868aa718-8104-4403-86dd-59b3c3940966","Type":"ContainerStarted","Data":"ac21068cef3ba061dcd476e6604b7ef87d032e70339bfb7dc7f8bff88ed6b3ef"} Oct 30 18:36:07.694543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:07.694491 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" podStartSLOduration=1.694477172 podStartE2EDuration="1.694477172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:07.691095576 +0000 UTC m=+2626.648099425" watchObservedRunningTime="2026-10-30 18:36:07.694477172 +0000 UTC m=+2626.651481007" Oct 30 18:36:07.751226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:07.751186 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:08.753621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:08.753578 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 18:36:08.754074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:08.753633 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" podUID="868aa718-8104-4403-86dd-59b3c3940966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 18:36:09.751760 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:09.751723 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 18:36:09.752089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:09.751792 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" podUID="868aa718-8104-4403-86dd-59b3c3940966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 18:36:10.751210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:10.751175 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 18:36:10.751590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:10.751229 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" podUID="868aa718-8104-4403-86dd-59b3c3940966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 18:36:11.794265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:11.784094 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 18:36:11.794265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:11.784197 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" podUID="868aa718-8104-4403-86dd-59b3c3940966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 18:36:12.751879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:12.751841 4883 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 18:36:12.752062 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:12.751897 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" podUID="868aa718-8104-4403-86dd-59b3c3940966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 18:36:13.767709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:13.767625 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:13.768279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:13.768159 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:13.768972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:13.768916 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:36:16.917823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:16.917787 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl"] Oct 30 18:36:16.922284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:16.922259 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:16.939285 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:16.939267 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:36:16.952795 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:16.952752 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:36:16.957903 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:16.957872 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl"] Oct 30 18:36:17.003987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.003959 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:36:17.004159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.004016 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:36:17.089339 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.089305 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl\" (UID: \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:17.089538 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.089444 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl\" (UID: \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:17.089634 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.089613 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8kd7\" (UniqueName: \"kubernetes.io/projected/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-kube-api-access-t8kd7\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl\" (UID: \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:17.193001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.191513 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl\" (UID: \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:17.193001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.191695 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kd7\" (UniqueName: \"kubernetes.io/projected/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-kube-api-access-t8kd7\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl\" (UID: \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:17.193001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.191979 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl\" (UID: \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:17.195482 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.194924 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl\" (UID: \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:17.196599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.196575 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl\" (UID: \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:17.226236 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.226205 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kd7\" (UniqueName: \"kubernetes.io/projected/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-kube-api-access-t8kd7\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl\" (UID: \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:17.238917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.238892 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:17.550167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.550138 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl"] Oct 30 18:36:17.551318 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:36:17.551284 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe840a22_b3e1_4d6b_a9f8_0e7469fcaecc.slice/crio-32d1281093c2f20023050c8b4ab0d12bd5488262ec756b8d24ff3bffdea3467d WatchSource:0}: Error finding container 32d1281093c2f20023050c8b4ab0d12bd5488262ec756b8d24ff3bffdea3467d: Status 404 returned error can't find the container with id 32d1281093c2f20023050c8b4ab0d12bd5488262ec756b8d24ff3bffdea3467d Oct 30 18:36:17.771466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:17.771432 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" event={"ID":"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc","Type":"ContainerStarted","Data":"32d1281093c2f20023050c8b4ab0d12bd5488262ec756b8d24ff3bffdea3467d"} Oct 30 18:36:18.788830 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:18.788731 4883 generic.go:356] "Generic (PLEG): container finished" podID="be840a22-b3e1-4d6b-a9f8-0e7469fcaecc" containerID="3b51d51cbf0b98396d4d0fab14434151ab4a56190598c67ec9d9377b1f398a08" exitCode=0 Oct 30 18:36:18.789340 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:18.788810 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" event={"ID":"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc","Type":"ContainerDied","Data":"3b51d51cbf0b98396d4d0fab14434151ab4a56190598c67ec9d9377b1f398a08"} Oct 30 18:36:18.983195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:18.983152 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq"] Oct 30 18:36:18.983638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:18.983355 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" podUID="3f40814a-3d5c-4e54-b5d8-8e9866d3b093" containerName="istio-proxy" containerID="cri-o://42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc" gracePeriod=30 Oct 30 18:36:19.974684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:19.974658 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:20.059877 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:20.059771 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-xks-ca\") pod \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\" (UID: \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\") " Oct 30 18:36:20.060112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:20.060003 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-openshift-ca\") pod \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\" (UID: \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\") " Oct 30 18:36:20.060199 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:20.060182 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-kube-api-access-t8kd7\" (UniqueName: \"kubernetes.io/projected/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-kube-api-access-t8kd7\") pod \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\" (UID: \"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc\") " Oct 30 18:36:20.060419 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:20.060391 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-openshift-ca" pod "be840a22-b3e1-4d6b-a9f8-0e7469fcaecc" (UID: "be840a22-b3e1-4d6b-a9f8-0e7469fcaecc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:20.060897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:20.060869 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:20.062267 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:20.062236 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-kube-api-access-t8kd7" pod "be840a22-b3e1-4d6b-a9f8-0e7469fcaecc" (UID: "be840a22-b3e1-4d6b-a9f8-0e7469fcaecc"). InnerVolumeSpecName "kube-api-access-t8kd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:20.062375 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:20.062273 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-xks-ca" pod "be840a22-b3e1-4d6b-a9f8-0e7469fcaecc" (UID: "be840a22-b3e1-4d6b-a9f8-0e7469fcaecc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:20.162632 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:20.162599 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8kd7\" (UniqueName: \"kubernetes.io/projected/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-kube-api-access-t8kd7\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:20.162632 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:20.162627 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:20.803026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:20.802996 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" Oct 30 18:36:20.803199 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:20.802989 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl" event={"ID":"be840a22-b3e1-4d6b-a9f8-0e7469fcaecc","Type":"ContainerDied","Data":"32d1281093c2f20023050c8b4ab0d12bd5488262ec756b8d24ff3bffdea3467d"} Oct 30 18:36:20.803199 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:20.803140 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32d1281093c2f20023050c8b4ab0d12bd5488262ec756b8d24ff3bffdea3467d" Oct 30 18:36:24.240632 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.240601 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:36:24.330030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.329909 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-token\" (UniqueName: \"kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-token\") pod \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " Oct 30 18:36:24.330231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.330071 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-data\") pod \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " Oct 30 18:36:24.330231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.330145 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istiod-ca-cert\") pod \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " Oct 30 18:36:24.330365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.330238 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-kube-api-access-jx47s\" (UniqueName: \"kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-kube-api-access-jx47s\") pod \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " Oct 30 18:36:24.330365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.330273 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-envoy\") pod \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " Oct 30 18:36:24.330365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.330349 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-certs\") pod \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " Oct 30 18:36:24.330533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.330383 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-socket\") pod \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " Oct 30 18:36:24.330533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.330489 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-credential-socket\") pod \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " Oct 30 18:36:24.330533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.330524 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-podinfo\") pod \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\" (UID: \"3f40814a-3d5c-4e54-b5d8-8e9866d3b093\") " Oct 30 18:36:24.333880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.331973 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-certs" pod "3f40814a-3d5c-4e54-b5d8-8e9866d3b093" (UID: "3f40814a-3d5c-4e54-b5d8-8e9866d3b093"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:24.333880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.332148 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-data" pod "3f40814a-3d5c-4e54-b5d8-8e9866d3b093" (UID: "3f40814a-3d5c-4e54-b5d8-8e9866d3b093"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:24.333880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.332464 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-socket" pod "3f40814a-3d5c-4e54-b5d8-8e9866d3b093" (UID: "3f40814a-3d5c-4e54-b5d8-8e9866d3b093"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:24.333880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.332647 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-credential-socket" pod "3f40814a-3d5c-4e54-b5d8-8e9866d3b093" (UID: "3f40814a-3d5c-4e54-b5d8-8e9866d3b093"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:24.333880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.332948 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istiod-ca-cert" pod "3f40814a-3d5c-4e54-b5d8-8e9866d3b093" (UID: "3f40814a-3d5c-4e54-b5d8-8e9866d3b093"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:24.334888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.334845 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-podinfo" pod "3f40814a-3d5c-4e54-b5d8-8e9866d3b093" (UID: "3f40814a-3d5c-4e54-b5d8-8e9866d3b093"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:36:24.345045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.338208 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-token" pod "3f40814a-3d5c-4e54-b5d8-8e9866d3b093" (UID: "3f40814a-3d5c-4e54-b5d8-8e9866d3b093"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:24.345045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.339799 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-envoy" pod "3f40814a-3d5c-4e54-b5d8-8e9866d3b093" (UID: "3f40814a-3d5c-4e54-b5d8-8e9866d3b093"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:24.345045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.340181 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-kube-api-access-jx47s" pod "3f40814a-3d5c-4e54-b5d8-8e9866d3b093" (UID: "3f40814a-3d5c-4e54-b5d8-8e9866d3b093"). InnerVolumeSpecName "kube-api-access-jx47s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:24.432598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.432567 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx47s\" (UniqueName: \"kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-kube-api-access-jx47s\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.432598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.432596 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.432598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.432606 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.432598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.432615 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.432875 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.432623 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.432875 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.432631 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.432875 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.432640 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.432875 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.432648 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.432875 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.432655 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f40814a-3d5c-4e54-b5d8-8e9866d3b093-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.834838 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.834808 4883 generic.go:356] "Generic (PLEG): container finished" podID="3f40814a-3d5c-4e54-b5d8-8e9866d3b093" containerID="42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc" exitCode=0 Oct 30 18:36:24.835137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.834863 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" Oct 30 18:36:24.835137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.834867 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" event={"ID":"3f40814a-3d5c-4e54-b5d8-8e9866d3b093","Type":"ContainerDied","Data":"42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc"} Oct 30 18:36:24.835137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.835100 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq" event={"ID":"3f40814a-3d5c-4e54-b5d8-8e9866d3b093","Type":"ContainerDied","Data":"a128a856318fceca6acb643e92d76345bb8c475df7e55067a49cd3b89c544887"} Oct 30 18:36:24.835137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.835132 4883 scope.go:122] "RemoveContainer" containerID="42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc" Oct 30 18:36:24.846758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.846741 4883 scope.go:122] "RemoveContainer" containerID="42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc" Oct 30 18:36:24.847039 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:36:24.847019 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc\": container with ID starting with 42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc not found: ID does not exist" containerID="42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc" Oct 30 18:36:24.847124 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.847044 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc"} err="failed to get container status \"42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc\": rpc error: code = NotFound desc = could not find container \"42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc\": container with ID starting with 42c9486616693599960b3b89e2fb6a4a523a40a6fa3cce3d6b142acb17db07cc not found: ID does not exist" Oct 30 18:36:24.894030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.893993 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq"] Oct 30 18:36:24.897832 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:24.897805 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ac632e-openshift-default-c98c999-hhlpq"] Oct 30 18:36:25.179174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:25.179141 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f40814a-3d5c-4e54-b5d8-8e9866d3b093" path="/var/lib/kubelet/pods/3f40814a-3d5c-4e54-b5d8-8e9866d3b093/volumes" Oct 30 18:36:29.633041 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.633002 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn"] Oct 30 18:36:29.633645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.633627 4883 cpu_manager.go:414] "containerMap: removing container" podUID="3f40814a-3d5c-4e54-b5d8-8e9866d3b093" containerName="istio-proxy" Oct 30 18:36:29.633645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.633644 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f40814a-3d5c-4e54-b5d8-8e9866d3b093" containerName="istio-proxy" Oct 30 18:36:29.633798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.633659 4883 cpu_manager.go:414] "containerMap: removing container" podUID="be840a22-b3e1-4d6b-a9f8-0e7469fcaecc" containerName="revoke-keys" Oct 30 18:36:29.633798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.633665 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be840a22-b3e1-4d6b-a9f8-0e7469fcaecc" containerName="revoke-keys" Oct 30 18:36:29.633798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.633751 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="be840a22-b3e1-4d6b-a9f8-0e7469fcaecc" containerName="revoke-keys" Oct 30 18:36:29.633798 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.633770 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f40814a-3d5c-4e54-b5d8-8e9866d3b093" containerName="istio-proxy" Oct 30 18:36:29.638589 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.638568 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:29.662566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.662535 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn"] Oct 30 18:36:29.669918 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.669896 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:36:29.672113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.670517 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:36:29.672113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.670584 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:36:29.672113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.671003 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:36:29.738989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.738954 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn\" (UID: \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:29.739182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.739017 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwldm\" (UniqueName: \"kubernetes.io/projected/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-kube-api-access-pwldm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn\" (UID: \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:29.739321 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.739291 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn\" (UID: \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:29.841442 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.841401 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn\" (UID: \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:29.841648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.841479 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwldm\" (UniqueName: \"kubernetes.io/projected/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-kube-api-access-pwldm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn\" (UID: \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:29.841648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.841639 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn\" (UID: \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:29.842806 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.842777 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn\" (UID: \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:29.863411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.863377 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn\" (UID: \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:29.875903 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.875863 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwldm\" (UniqueName: \"kubernetes.io/projected/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-kube-api-access-pwldm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn\" (UID: \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:29.952188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:29.952117 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:30.265316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:30.265287 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn"] Oct 30 18:36:30.266458 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:36:30.266427 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe693e2_e6cc_40ea_a78d_0bfd0f62db72.slice/crio-653ada95f50ebe6e66b072ef6b895ac43974a62c449a2c51ead4826b0adab3d1 WatchSource:0}: Error finding container 653ada95f50ebe6e66b072ef6b895ac43974a62c449a2c51ead4826b0adab3d1: Status 404 returned error can't find the container with id 653ada95f50ebe6e66b072ef6b895ac43974a62c449a2c51ead4826b0adab3d1 Oct 30 18:36:30.896475 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:30.896433 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" event={"ID":"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72","Type":"ContainerStarted","Data":"ee0819f6984ccd1758986bea1617dbe182c547a4918bb45c3aff6ee628a1b6d5"} Oct 30 18:36:30.897047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:30.896501 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" event={"ID":"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72","Type":"ContainerStarted","Data":"653ada95f50ebe6e66b072ef6b895ac43974a62c449a2c51ead4826b0adab3d1"} Oct 30 18:36:30.940826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:30.940773 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" podStartSLOduration=1.940758199 podStartE2EDuration="1.940758199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:30.939056585 +0000 UTC m=+2649.896060419" watchObservedRunningTime="2026-10-30 18:36:30.940758199 +0000 UTC m=+2649.897762033" Oct 30 18:36:31.904856 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:31.904826 4883 generic.go:356] "Generic (PLEG): container finished" podID="ffe693e2-e6cc-40ea-a78d-0bfd0f62db72" containerID="ee0819f6984ccd1758986bea1617dbe182c547a4918bb45c3aff6ee628a1b6d5" exitCode=0 Oct 30 18:36:31.905302 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:31.904912 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" event={"ID":"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72","Type":"ContainerDied","Data":"ee0819f6984ccd1758986bea1617dbe182c547a4918bb45c3aff6ee628a1b6d5"} Oct 30 18:36:33.079313 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.079286 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:33.194271 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.194245 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-openshift-ca\") pod \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\" (UID: \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\") " Oct 30 18:36:33.194417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.194290 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-xks-ca\") pod \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\" (UID: \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\") " Oct 30 18:36:33.194417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.194332 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-kube-api-access-pwldm\" (UniqueName: \"kubernetes.io/projected/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-kube-api-access-pwldm\") pod \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\" (UID: \"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72\") " Oct 30 18:36:33.194597 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.194576 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-openshift-ca" pod "ffe693e2-e6cc-40ea-a78d-0bfd0f62db72" (UID: "ffe693e2-e6cc-40ea-a78d-0bfd0f62db72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:33.194917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.194899 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:33.196438 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.196416 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-xks-ca" pod "ffe693e2-e6cc-40ea-a78d-0bfd0f62db72" (UID: "ffe693e2-e6cc-40ea-a78d-0bfd0f62db72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:33.196516 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.196440 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-kube-api-access-pwldm" pod "ffe693e2-e6cc-40ea-a78d-0bfd0f62db72" (UID: "ffe693e2-e6cc-40ea-a78d-0bfd0f62db72"). InnerVolumeSpecName "kube-api-access-pwldm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:33.296133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.296100 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:33.296133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.296130 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwldm\" (UniqueName: \"kubernetes.io/projected/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72-kube-api-access-pwldm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:33.921208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.921167 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" event={"ID":"ffe693e2-e6cc-40ea-a78d-0bfd0f62db72","Type":"ContainerDied","Data":"653ada95f50ebe6e66b072ef6b895ac43974a62c449a2c51ead4826b0adab3d1"} Oct 30 18:36:33.921399 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.921181 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn" Oct 30 18:36:33.921399 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:33.921267 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="653ada95f50ebe6e66b072ef6b895ac43974a62c449a2c51ead4826b0adab3d1" Oct 30 18:36:35.082022 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:36:35.081984 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:36:36.685384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:36.685349 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw"] Oct 30 18:36:37.722452 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:37.722411 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7"] Oct 30 18:36:51.727259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:51.727177 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" containerID="cri-o://c4c0db54b5fbd13b11bb562717315cab1f9512523193995adab4f166d40251b8" gracePeriod=45 Oct 30 18:36:52.096965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.094331 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": dial tcp 10.132.0.198:8000: connect: connection refused" Oct 30 18:36:52.220018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.219887 4883 generic.go:356] "Generic (PLEG): container finished" podID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerID="c4c0db54b5fbd13b11bb562717315cab1f9512523193995adab4f166d40251b8" exitCode=0 Oct 30 18:36:52.220018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.219975 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" event={"ID":"e5d98758-4cc4-49e2-9551-34b1a9be56dc","Type":"ContainerDied","Data":"c4c0db54b5fbd13b11bb562717315cab1f9512523193995adab4f166d40251b8"} Oct 30 18:36:52.430652 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.430627 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:36:52.556570 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.556539 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl"] Oct 30 18:36:52.559854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.559831 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a2a10a-4xjtl"] Oct 30 18:36:52.605073 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.605037 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tls-certs\") pod \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " Oct 30 18:36:52.605249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.605111 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d98758-4cc4-49e2-9551-34b1a9be56dc-kube-api-access-bsbj4\" (UniqueName: \"kubernetes.io/projected/e5d98758-4cc4-49e2-9551-34b1a9be56dc-kube-api-access-bsbj4\") pod \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " Oct 30 18:36:52.605249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.605154 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-home\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-home\") pod \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " Oct 30 18:36:52.605249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.605179 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-model-cache\") pod \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " Oct 30 18:36:52.605425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.605258 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tmp-dir\") pod \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " Oct 30 18:36:52.605425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.605338 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-dshm\") pod \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\" (UID: \"e5d98758-4cc4-49e2-9551-34b1a9be56dc\") " Oct 30 18:36:52.605528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.605496 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-home" pod "e5d98758-4cc4-49e2-9551-34b1a9be56dc" (UID: "e5d98758-4cc4-49e2-9551-34b1a9be56dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:52.605528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.605507 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-model-cache" pod "e5d98758-4cc4-49e2-9551-34b1a9be56dc" (UID: "e5d98758-4cc4-49e2-9551-34b1a9be56dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:52.605890 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.605858 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tmp-dir" pod "e5d98758-4cc4-49e2-9551-34b1a9be56dc" (UID: "e5d98758-4cc4-49e2-9551-34b1a9be56dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:52.606133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.606113 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:52.606188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.606138 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:52.606188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.606154 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:52.608051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.608023 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d98758-4cc4-49e2-9551-34b1a9be56dc-kube-api-access-bsbj4" pod "e5d98758-4cc4-49e2-9551-34b1a9be56dc" (UID: "e5d98758-4cc4-49e2-9551-34b1a9be56dc"). InnerVolumeSpecName "kube-api-access-bsbj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:52.608155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.608064 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tls-certs" pod "e5d98758-4cc4-49e2-9551-34b1a9be56dc" (UID: "e5d98758-4cc4-49e2-9551-34b1a9be56dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:52.608155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.608069 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-dshm" pod "e5d98758-4cc4-49e2-9551-34b1a9be56dc" (UID: "e5d98758-4cc4-49e2-9551-34b1a9be56dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:52.711817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.711697 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d98758-4cc4-49e2-9551-34b1a9be56dc-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:52.711817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.711742 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsbj4\" (UniqueName: \"kubernetes.io/projected/e5d98758-4cc4-49e2-9551-34b1a9be56dc-kube-api-access-bsbj4\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:52.711817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.711758 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d98758-4cc4-49e2-9551-34b1a9be56dc-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:52.745375 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.745333 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" containerID="cri-o://c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691" gracePeriod=45 Oct 30 18:36:52.825008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:52.824974 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": dial tcp 10.132.0.199:8000: connect: connection refused" Oct 30 18:36:53.050926 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.050897 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:36:53.116808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.116644 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78f62c40-cbfc-42da-8210-5f64075a3e25-tls-certs\" (UniqueName: \"kubernetes.io/secret/78f62c40-cbfc-42da-8210-5f64075a3e25-tls-certs\") pod \"78f62c40-cbfc-42da-8210-5f64075a3e25\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " Oct 30 18:36:53.116808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.116701 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78f62c40-cbfc-42da-8210-5f64075a3e25-kube-api-access-zbhjl\" (UniqueName: \"kubernetes.io/projected/78f62c40-cbfc-42da-8210-5f64075a3e25-kube-api-access-zbhjl\") pod \"78f62c40-cbfc-42da-8210-5f64075a3e25\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " Oct 30 18:36:53.116808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.116803 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-tmp-dir\") pod \"78f62c40-cbfc-42da-8210-5f64075a3e25\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " Oct 30 18:36:53.117142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.117106 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-tmp-dir" pod "78f62c40-cbfc-42da-8210-5f64075a3e25" (UID: "78f62c40-cbfc-42da-8210-5f64075a3e25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:53.117303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.117282 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-model-cache\") pod \"78f62c40-cbfc-42da-8210-5f64075a3e25\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " Oct 30 18:36:53.117395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.117371 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-dshm\") pod \"78f62c40-cbfc-42da-8210-5f64075a3e25\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " Oct 30 18:36:53.117475 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.117455 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-home\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-home\") pod \"78f62c40-cbfc-42da-8210-5f64075a3e25\" (UID: \"78f62c40-cbfc-42da-8210-5f64075a3e25\") " Oct 30 18:36:53.117602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.117575 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-model-cache" pod "78f62c40-cbfc-42da-8210-5f64075a3e25" (UID: "78f62c40-cbfc-42da-8210-5f64075a3e25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:53.117750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.117727 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-home" pod "78f62c40-cbfc-42da-8210-5f64075a3e25" (UID: "78f62c40-cbfc-42da-8210-5f64075a3e25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:53.118292 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.118267 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:53.118292 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.118292 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:53.118454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.118300 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:53.119650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.119619 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f62c40-cbfc-42da-8210-5f64075a3e25-kube-api-access-zbhjl" pod "78f62c40-cbfc-42da-8210-5f64075a3e25" (UID: "78f62c40-cbfc-42da-8210-5f64075a3e25"). InnerVolumeSpecName "kube-api-access-zbhjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:53.119650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.119641 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78f62c40-cbfc-42da-8210-5f64075a3e25-tls-certs" pod "78f62c40-cbfc-42da-8210-5f64075a3e25" (UID: "78f62c40-cbfc-42da-8210-5f64075a3e25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:53.120247 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.120215 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-dshm" pod "78f62c40-cbfc-42da-8210-5f64075a3e25" (UID: "78f62c40-cbfc-42da-8210-5f64075a3e25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:53.178640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.178607 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be840a22-b3e1-4d6b-a9f8-0e7469fcaecc" path="/var/lib/kubelet/pods/be840a22-b3e1-4d6b-a9f8-0e7469fcaecc/volumes" Oct 30 18:36:53.219578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.219549 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78f62c40-cbfc-42da-8210-5f64075a3e25-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:53.219578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.219580 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbhjl\" (UniqueName: \"kubernetes.io/projected/78f62c40-cbfc-42da-8210-5f64075a3e25-kube-api-access-zbhjl\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:53.219741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.219593 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78f62c40-cbfc-42da-8210-5f64075a3e25-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:36:53.228050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.228026 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" Oct 30 18:36:53.228174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.228055 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw" event={"ID":"e5d98758-4cc4-49e2-9551-34b1a9be56dc","Type":"ContainerDied","Data":"5095bd7aaa0adb53832151a1b7f760ac647a7185c0aa94dd9f9386fd91d3aa6b"} Oct 30 18:36:53.228174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.228111 4883 scope.go:122] "RemoveContainer" containerID="c4c0db54b5fbd13b11bb562717315cab1f9512523193995adab4f166d40251b8" Oct 30 18:36:53.229524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.229502 4883 generic.go:356] "Generic (PLEG): container finished" podID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerID="c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691" exitCode=0 Oct 30 18:36:53.229622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.229541 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" Oct 30 18:36:53.229622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.229574 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" event={"ID":"78f62c40-cbfc-42da-8210-5f64075a3e25","Type":"ContainerDied","Data":"c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691"} Oct 30 18:36:53.229622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.229603 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7" event={"ID":"78f62c40-cbfc-42da-8210-5f64075a3e25","Type":"ContainerDied","Data":"d4d9f9cfdfb99535fd8c03ef14c556415e1938f06c42106cc070b7614e6a49bd"} Oct 30 18:36:53.238127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.238112 4883 scope.go:122] "RemoveContainer" containerID="c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691" Oct 30 18:36:53.250530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.250513 4883 scope.go:122] "RemoveContainer" containerID="c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691" Oct 30 18:36:53.250848 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:36:53.250819 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691\": container with ID starting with c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691 not found: ID does not exist" containerID="c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691" Oct 30 18:36:53.250956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.250854 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691"} err="failed to get container status \"c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691\": rpc error: code = NotFound desc = could not find container \"c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691\": container with ID starting with c6086f4465aa8d638feb767cb2a13b85f9ffc0bbe1a8b0b828670a9d296ed691 not found: ID does not exist" Oct 30 18:36:53.287698 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.287617 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw"] Oct 30 18:36:53.292320 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.292286 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e77b1d/auth-test-model-e77b1d-kserve-6f66f4b54f-6qwvw"] Oct 30 18:36:53.326107 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.326076 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7"] Oct 30 18:36:53.335144 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:53.335120 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-50e9fc/auth-test-model-50e9fc-kserve-5f4779568f-4w4t7"] Oct 30 18:36:55.181648 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.181615 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" path="/var/lib/kubelet/pods/78f62c40-cbfc-42da-8210-5f64075a3e25/volumes" Oct 30 18:36:55.182358 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.182333 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" path="/var/lib/kubelet/pods/e5d98758-4cc4-49e2-9551-34b1a9be56dc/volumes" Oct 30 18:36:55.875452 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.875415 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck"] Oct 30 18:36:55.876123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.876102 4883 cpu_manager.go:414] "containerMap: removing container" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" Oct 30 18:36:55.876123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.876122 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" Oct 30 18:36:55.876301 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.876147 4883 cpu_manager.go:414] "containerMap: removing container" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" Oct 30 18:36:55.876301 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.876154 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" Oct 30 18:36:55.876301 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.876176 4883 cpu_manager.go:414] "containerMap: removing container" podUID="ffe693e2-e6cc-40ea-a78d-0bfd0f62db72" containerName="revoke-keys" Oct 30 18:36:55.876301 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.876184 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe693e2-e6cc-40ea-a78d-0bfd0f62db72" containerName="revoke-keys" Oct 30 18:36:55.876503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.876343 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe693e2-e6cc-40ea-a78d-0bfd0f62db72" containerName="revoke-keys" Oct 30 18:36:55.876503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.876369 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d98758-4cc4-49e2-9551-34b1a9be56dc" containerName="main" Oct 30 18:36:55.876503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.876381 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="78f62c40-cbfc-42da-8210-5f64075a3e25" containerName="main" Oct 30 18:36:55.884991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.881860 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:55.913656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.913628 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck"] Oct 30 18:36:55.917684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:55.917092 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-a1a31f-openshift-default-dockercfg-g2jkh\"" Oct 30 18:36:56.011687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.011651 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxqqz\" (UniqueName: \"kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-kube-api-access-hxqqz\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.011879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.011726 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-certs\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.011879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.011766 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-credential-socket\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.011993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.011904 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-envoy\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.012077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.012056 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istiod-ca-cert\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.012176 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.012158 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-podinfo\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.012216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.012193 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-socket\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.012268 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.012222 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-token\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.012369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.012355 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-data\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.116966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.116383 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-envoy\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.116966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.116495 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istiod-ca-cert\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.116966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.116593 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-podinfo\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.116966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.116622 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-socket\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.116966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.116661 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-token\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.116966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.116743 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-data\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.116966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.116796 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqqz\" (UniqueName: \"kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-kube-api-access-hxqqz\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.116966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.116853 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-certs\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.116966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.116889 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-credential-socket\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.117681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.117387 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-credential-socket\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.126419 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.126338 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-envoy\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.130954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.127136 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istiod-ca-cert\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.142002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.135364 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-podinfo\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.142002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.135625 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-socket\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.142002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.136154 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-data\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.142002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.136654 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-certs\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.176963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.170619 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-token\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.192965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.179585 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxqqz\" (UniqueName: \"kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-kube-api-access-hxqqz\") pod \"e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.211423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.210219 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:56.713358 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.713321 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck"] Oct 30 18:36:56.714826 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:36:56.714783 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2ea73e_ecbd_4e79_8e24_d6e2a5ef3470.slice/crio-e50a02a9e4a34ebbb25bf5decf255d3d3f64b569423dab4661a63c06892874eb WatchSource:0}: Error finding container e50a02a9e4a34ebbb25bf5decf255d3d3f64b569423dab4661a63c06892874eb: Status 404 returned error can't find the container with id e50a02a9e4a34ebbb25bf5decf255d3d3f64b569423dab4661a63c06892874eb Oct 30 18:36:56.717701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.717667 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:36:56.717782 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.717736 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:36:56.717782 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:56.717767 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:36:57.403834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:57.402455 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" event={"ID":"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470","Type":"ContainerStarted","Data":"20da60c9571f2a60babdc1170cc039a0f239d2b0f5e116c2bb210a06900a642d"} Oct 30 18:36:57.403834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:57.402525 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" event={"ID":"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470","Type":"ContainerStarted","Data":"e50a02a9e4a34ebbb25bf5decf255d3d3f64b569423dab4661a63c06892874eb"} Oct 30 18:36:57.589497 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:57.589432 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" podStartSLOduration=2.5894114310000003 podStartE2EDuration="2.589411431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:57.480613575 +0000 UTC m=+2676.437617425" watchObservedRunningTime="2026-10-30 18:36:57.589411431 +0000 UTC m=+2676.546415267" Oct 30 18:36:58.163998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:58.162903 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn"] Oct 30 18:36:58.195957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:58.195070 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5cb3d5-lmjxn"] Oct 30 18:36:58.225771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:58.221010 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:58.245746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:58.245718 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:58.409296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:58.409267 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:58.410103 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:58.410084 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:36:59.185369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:36:59.185336 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe693e2-e6cc-40ea-a78d-0bfd0f62db72" path="/var/lib/kubelet/pods/ffe693e2-e6cc-40ea-a78d-0bfd0f62db72/volumes" Oct 30 18:37:00.403526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:00.403486 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk"] Oct 30 18:37:00.404065 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:00.403719 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" podUID="868aa718-8104-4403-86dd-59b3c3940966" containerName="istio-proxy" containerID="cri-o://0ce268a1d2d3533bd456a6f52d27aa2437095ef171e6ed4446c830f39e874a81" gracePeriod=30 Oct 30 18:37:00.414994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:00.414951 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8"] Oct 30 18:37:00.415250 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:00.415205 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" podUID="56f24af4-568d-4e91-8458-79fad2821ad3" containerName="istio-proxy" containerID="cri-o://db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75" gracePeriod=30 Oct 30 18:37:00.956989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:00.956948 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq"] Oct 30 18:37:00.962582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:00.962560 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:00.980050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:00.980026 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:37:00.980212 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:00.980115 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:37:00.980337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:00.980229 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:37:00.980432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:00.980352 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:37:00.984981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:00.984956 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq"] Oct 30 18:37:01.075168 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:01.075139 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq\" (UID: \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:01.075470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:01.075446 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq\" (UID: \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:01.075579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:01.075563 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx9nn\" (UniqueName: \"kubernetes.io/projected/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-kube-api-access-vx9nn\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq\" (UID: \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:01.176708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:01.176670 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq\" (UID: \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:01.176889 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:01.176762 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx9nn\" (UniqueName: \"kubernetes.io/projected/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-kube-api-access-vx9nn\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq\" (UID: \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:01.176889 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:01.176840 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq\" (UID: \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:01.177503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:01.177469 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq\" (UID: \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:01.179098 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:01.179079 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq\" (UID: \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:01.201370 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:01.201347 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx9nn\" (UniqueName: \"kubernetes.io/projected/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-kube-api-access-vx9nn\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq\" (UID: \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:01.277769 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:01.277707 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:01.586238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:01.586211 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq"] Oct 30 18:37:01.587149 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:37:01.587123 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ee8d4d_c26b_4cae_b0ef_5f0e55b9e5db.slice/crio-d1cdc595f3a2c2a40e21137d4e8b134c81ac07505989b1756765804d9d550fac WatchSource:0}: Error finding container d1cdc595f3a2c2a40e21137d4e8b134c81ac07505989b1756765804d9d550fac: Status 404 returned error can't find the container with id d1cdc595f3a2c2a40e21137d4e8b134c81ac07505989b1756765804d9d550fac Oct 30 18:37:02.511957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:02.505714 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" event={"ID":"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db","Type":"ContainerStarted","Data":"ec7f2709047e35cfc7e5893d3b1d25ce4c1ad3bd6648299cacebb0d82ca39fa5"} Oct 30 18:37:02.511957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:02.505784 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" event={"ID":"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db","Type":"ContainerStarted","Data":"d1cdc595f3a2c2a40e21137d4e8b134c81ac07505989b1756765804d9d550fac"} Oct 30 18:37:02.558693 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:02.558642 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" podStartSLOduration=2.558627497 podStartE2EDuration="2.558627497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:02.557735442 +0000 UTC m=+2681.514739277" watchObservedRunningTime="2026-10-30 18:37:02.558627497 +0000 UTC m=+2681.515631332" Oct 30 18:37:03.514122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:03.514076 4883 generic.go:356] "Generic (PLEG): container finished" podID="f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db" containerID="ec7f2709047e35cfc7e5893d3b1d25ce4c1ad3bd6648299cacebb0d82ca39fa5" exitCode=0 Oct 30 18:37:03.514517 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:03.514160 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" event={"ID":"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db","Type":"ContainerDied","Data":"ec7f2709047e35cfc7e5893d3b1d25ce4c1ad3bd6648299cacebb0d82ca39fa5"} Oct 30 18:37:04.709595 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:04.709571 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:04.751790 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:04.751755 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-kube-api-access-vx9nn\" (UniqueName: \"kubernetes.io/projected/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-kube-api-access-vx9nn\") pod \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\" (UID: \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\") " Oct 30 18:37:04.751968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:04.751847 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-openshift-ca\") pod \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\" (UID: \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\") " Oct 30 18:37:04.752021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:04.751975 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-xks-ca\") pod \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\" (UID: \"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db\") " Oct 30 18:37:04.752268 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:04.752238 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-openshift-ca" pod "f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db" (UID: "f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:04.752621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:04.752605 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:04.753978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:04.753946 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-xks-ca" pod "f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db" (UID: "f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:04.754091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:04.753975 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-kube-api-access-vx9nn" pod "f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db" (UID: "f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db"). InnerVolumeSpecName "kube-api-access-vx9nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:04.853521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:04.853490 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx9nn\" (UniqueName: \"kubernetes.io/projected/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-kube-api-access-vx9nn\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:04.853521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:04.853519 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.078662 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:37:05.078627 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:37:05.530753 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.530725 4883 generic.go:356] "Generic (PLEG): container finished" podID="868aa718-8104-4403-86dd-59b3c3940966" containerID="0ce268a1d2d3533bd456a6f52d27aa2437095ef171e6ed4446c830f39e874a81" exitCode=0 Oct 30 18:37:05.530863 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.530793 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" event={"ID":"868aa718-8104-4403-86dd-59b3c3940966","Type":"ContainerDied","Data":"0ce268a1d2d3533bd456a6f52d27aa2437095ef171e6ed4446c830f39e874a81"} Oct 30 18:37:05.532345 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.532324 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" event={"ID":"f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db","Type":"ContainerDied","Data":"d1cdc595f3a2c2a40e21137d4e8b134c81ac07505989b1756765804d9d550fac"} Oct 30 18:37:05.532447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.532342 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq" Oct 30 18:37:05.532447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.532367 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1cdc595f3a2c2a40e21137d4e8b134c81ac07505989b1756765804d9d550fac" Oct 30 18:37:05.799414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.799386 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:37:05.826037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.826015 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:37:05.885414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885381 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/868aa718-8104-4403-86dd-59b3c3940966-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868aa718-8104-4403-86dd-59b3c3940966-istiod-ca-cert\") pod \"868aa718-8104-4403-86dd-59b3c3940966\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " Oct 30 18:37:05.885588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885456 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-istio-token\" (UniqueName: \"kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-istio-token\") pod \"868aa718-8104-4403-86dd-59b3c3940966\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " Oct 30 18:37:05.885588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885491 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56f24af4-568d-4e91-8458-79fad2821ad3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f24af4-568d-4e91-8458-79fad2821ad3-istio-podinfo\") pod \"56f24af4-568d-4e91-8458-79fad2821ad3\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " Oct 30 18:37:05.885588 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885543 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-envoy\") pod \"56f24af4-568d-4e91-8458-79fad2821ad3\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " Oct 30 18:37:05.885764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885617 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-certs\") pod \"868aa718-8104-4403-86dd-59b3c3940966\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " Oct 30 18:37:05.885764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885665 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-envoy\") pod \"868aa718-8104-4403-86dd-59b3c3940966\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " Oct 30 18:37:05.885764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885717 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-socket\") pod \"868aa718-8104-4403-86dd-59b3c3940966\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " Oct 30 18:37:05.885764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885740 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-credential-socket\") pod \"868aa718-8104-4403-86dd-59b3c3940966\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " Oct 30 18:37:05.885764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885763 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-kube-api-access-86b8h\" (UniqueName: \"kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-kube-api-access-86b8h\") pod \"868aa718-8104-4403-86dd-59b3c3940966\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " Oct 30 18:37:05.885998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885822 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f24af4-568d-4e91-8458-79fad2821ad3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f24af4-568d-4e91-8458-79fad2821ad3-istiod-ca-cert\") pod \"56f24af4-568d-4e91-8458-79fad2821ad3\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " Oct 30 18:37:05.885998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885848 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-credential-socket\") pod \"56f24af4-568d-4e91-8458-79fad2821ad3\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " Oct 30 18:37:05.885998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885864 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-data\") pod \"868aa718-8104-4403-86dd-59b3c3940966\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " Oct 30 18:37:05.885998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885897 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-istio-token\" (UniqueName: \"kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-istio-token\") pod \"56f24af4-568d-4e91-8458-79fad2821ad3\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " Oct 30 18:37:05.885998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885961 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-data\") pod \"56f24af4-568d-4e91-8458-79fad2821ad3\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " Oct 30 18:37:05.885998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885990 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/868aa718-8104-4403-86dd-59b3c3940966-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868aa718-8104-4403-86dd-59b3c3940966-istio-podinfo\") pod \"868aa718-8104-4403-86dd-59b3c3940966\" (UID: \"868aa718-8104-4403-86dd-59b3c3940966\") " Oct 30 18:37:05.886208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.886021 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-socket\") pod \"56f24af4-568d-4e91-8458-79fad2821ad3\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " Oct 30 18:37:05.886208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.886042 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-kube-api-access-qwlbf\" (UniqueName: \"kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-kube-api-access-qwlbf\") pod \"56f24af4-568d-4e91-8458-79fad2821ad3\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " Oct 30 18:37:05.886208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.886064 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-certs\") pod \"56f24af4-568d-4e91-8458-79fad2821ad3\" (UID: \"56f24af4-568d-4e91-8458-79fad2821ad3\") " Oct 30 18:37:05.887268 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.885815 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868aa718-8104-4403-86dd-59b3c3940966-istiod-ca-cert" pod "868aa718-8104-4403-86dd-59b3c3940966" (UID: "868aa718-8104-4403-86dd-59b3c3940966"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:05.887386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.886143 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-certs" pod "868aa718-8104-4403-86dd-59b3c3940966" (UID: "868aa718-8104-4403-86dd-59b3c3940966"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:05.887386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.886543 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-credential-socket" pod "56f24af4-568d-4e91-8458-79fad2821ad3" (UID: "56f24af4-568d-4e91-8458-79fad2821ad3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:05.887386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.886677 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-credential-socket" pod "868aa718-8104-4403-86dd-59b3c3940966" (UID: "868aa718-8104-4403-86dd-59b3c3940966"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:05.887386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.887009 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f24af4-568d-4e91-8458-79fad2821ad3-istiod-ca-cert" pod "56f24af4-568d-4e91-8458-79fad2821ad3" (UID: "56f24af4-568d-4e91-8458-79fad2821ad3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:05.887386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.887030 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-socket" pod "868aa718-8104-4403-86dd-59b3c3940966" (UID: "868aa718-8104-4403-86dd-59b3c3940966"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:05.887386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.887234 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-data" pod "868aa718-8104-4403-86dd-59b3c3940966" (UID: "868aa718-8104-4403-86dd-59b3c3940966"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:05.887386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.887245 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-data" pod "56f24af4-568d-4e91-8458-79fad2821ad3" (UID: "56f24af4-568d-4e91-8458-79fad2821ad3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:05.887777 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.887500 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-socket" pod "56f24af4-568d-4e91-8458-79fad2821ad3" (UID: "56f24af4-568d-4e91-8458-79fad2821ad3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:05.887777 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.887609 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-certs" pod "56f24af4-568d-4e91-8458-79fad2821ad3" (UID: "56f24af4-568d-4e91-8458-79fad2821ad3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:05.888687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.888660 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-istio-token" pod "868aa718-8104-4403-86dd-59b3c3940966" (UID: "868aa718-8104-4403-86dd-59b3c3940966"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:05.889774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.889751 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56f24af4-568d-4e91-8458-79fad2821ad3-istio-podinfo" pod "56f24af4-568d-4e91-8458-79fad2821ad3" (UID: "56f24af4-568d-4e91-8458-79fad2821ad3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:05.889995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.889973 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-envoy" pod "868aa718-8104-4403-86dd-59b3c3940966" (UID: "868aa718-8104-4403-86dd-59b3c3940966"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:05.890094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.890005 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-envoy" pod "56f24af4-568d-4e91-8458-79fad2821ad3" (UID: "56f24af4-568d-4e91-8458-79fad2821ad3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:05.890094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.890066 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/868aa718-8104-4403-86dd-59b3c3940966-istio-podinfo" pod "868aa718-8104-4403-86dd-59b3c3940966" (UID: "868aa718-8104-4403-86dd-59b3c3940966"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:05.890317 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.890292 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-kube-api-access-86b8h" pod "868aa718-8104-4403-86dd-59b3c3940966" (UID: "868aa718-8104-4403-86dd-59b3c3940966"). InnerVolumeSpecName "kube-api-access-86b8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:05.890624 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.890602 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-kube-api-access-qwlbf" pod "56f24af4-568d-4e91-8458-79fad2821ad3" (UID: "56f24af4-568d-4e91-8458-79fad2821ad3"). InnerVolumeSpecName "kube-api-access-qwlbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:05.891082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.891065 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-istio-token" pod "56f24af4-568d-4e91-8458-79fad2821ad3" (UID: "56f24af4-568d-4e91-8458-79fad2821ad3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:05.988313 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988237 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988313 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988275 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988313 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988288 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988313 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988300 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988313 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988313 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988332 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86b8h\" (UniqueName: \"kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-kube-api-access-86b8h\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988345 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f24af4-568d-4e91-8458-79fad2821ad3-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988360 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988372 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868aa718-8104-4403-86dd-59b3c3940966-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988386 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988398 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988410 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868aa718-8104-4403-86dd-59b3c3940966-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988422 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988437 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwlbf\" (UniqueName: \"kubernetes.io/projected/56f24af4-568d-4e91-8458-79fad2821ad3-kube-api-access-qwlbf\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988450 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f24af4-568d-4e91-8458-79fad2821ad3-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988462 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868aa718-8104-4403-86dd-59b3c3940966-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988473 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868aa718-8104-4403-86dd-59b3c3940966-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:05.988603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:05.988487 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f24af4-568d-4e91-8458-79fad2821ad3-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:06.548474 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.548423 4883 generic.go:356] "Generic (PLEG): container finished" podID="56f24af4-568d-4e91-8458-79fad2821ad3" containerID="db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75" exitCode=0 Oct 30 18:37:06.548681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.548518 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" event={"ID":"56f24af4-568d-4e91-8458-79fad2821ad3","Type":"ContainerDied","Data":"db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75"} Oct 30 18:37:06.548681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.548571 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" Oct 30 18:37:06.548681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.548619 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8" event={"ID":"56f24af4-568d-4e91-8458-79fad2821ad3","Type":"ContainerDied","Data":"21e0f0b7d8ddd1131f517208e544fb25d986da4c6c8c306259b5f5a49248b8ad"} Oct 30 18:37:06.548681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.548671 4883 scope.go:122] "RemoveContainer" containerID="db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75" Oct 30 18:37:06.551074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.551056 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" Oct 30 18:37:06.551197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.551103 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk" event={"ID":"868aa718-8104-4403-86dd-59b3c3940966","Type":"ContainerDied","Data":"ac21068cef3ba061dcd476e6604b7ef87d032e70339bfb7dc7f8bff88ed6b3ef"} Oct 30 18:37:06.561159 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.561138 4883 scope.go:122] "RemoveContainer" containerID="db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75" Oct 30 18:37:06.561442 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:37:06.561419 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75\": container with ID starting with db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75 not found: ID does not exist" containerID="db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75" Oct 30 18:37:06.561528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.561454 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75"} err="failed to get container status \"db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75\": rpc error: code = NotFound desc = could not find container \"db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75\": container with ID starting with db9e328c81972c611ce2d20d94f11efcb3143db8849f689649d53f2a08f26b75 not found: ID does not exist" Oct 30 18:37:06.561528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.561480 4883 scope.go:122] "RemoveContainer" containerID="0ce268a1d2d3533bd456a6f52d27aa2437095ef171e6ed4446c830f39e874a81" Oct 30 18:37:06.608585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.608562 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk"] Oct 30 18:37:06.621447 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.621425 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-5rhfk"] Oct 30 18:37:06.650022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.649993 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8"] Oct 30 18:37:06.653556 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:06.653534 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a2a10a-openshift-default-5d66c5686c-25ks8"] Oct 30 18:37:07.180066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:07.180033 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f24af4-568d-4e91-8458-79fad2821ad3" path="/var/lib/kubelet/pods/56f24af4-568d-4e91-8458-79fad2821ad3/volumes" Oct 30 18:37:07.180531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:07.180517 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="868aa718-8104-4403-86dd-59b3c3940966" path="/var/lib/kubelet/pods/868aa718-8104-4403-86dd-59b3c3940966/volumes" Oct 30 18:37:21.926371 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:21.926339 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:37:21.927739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:21.927710 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:37:21.933602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:21.933569 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:37:21.934610 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:21.934592 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:37:26.420997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:26.420316 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9"] Oct 30 18:37:26.420997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:26.420562 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerName="istio-proxy" containerID="cri-o://a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb" gracePeriod=30 Oct 30 18:37:27.041013 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.040973 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5"] Oct 30 18:37:27.052525 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.052487 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d21505d3-c5rl5"] Oct 30 18:37:27.215206 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.215166 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2ecef6-4d24-4410-b822-ce7a03f6b2e0" path="/var/lib/kubelet/pods/cc2ecef6-4d24-4410-b822-ce7a03f6b2e0/volumes" Oct 30 18:37:27.291221 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.291133 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9"] Oct 30 18:37:27.291880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.291861 4883 cpu_manager.go:414] "containerMap: removing container" podUID="f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db" containerName="revoke-keys" Oct 30 18:37:27.291880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.291880 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db" containerName="revoke-keys" Oct 30 18:37:27.292045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.291908 4883 cpu_manager.go:414] "containerMap: removing container" podUID="868aa718-8104-4403-86dd-59b3c3940966" containerName="istio-proxy" Oct 30 18:37:27.292045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.291913 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868aa718-8104-4403-86dd-59b3c3940966" containerName="istio-proxy" Oct 30 18:37:27.292045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.291929 4883 cpu_manager.go:414] "containerMap: removing container" podUID="56f24af4-568d-4e91-8458-79fad2821ad3" containerName="istio-proxy" Oct 30 18:37:27.292045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.291948 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f24af4-568d-4e91-8458-79fad2821ad3" containerName="istio-proxy" Oct 30 18:37:27.292171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.292062 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db" containerName="revoke-keys" Oct 30 18:37:27.292171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.292077 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="868aa718-8104-4403-86dd-59b3c3940966" containerName="istio-proxy" Oct 30 18:37:27.292171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.292084 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f24af4-568d-4e91-8458-79fad2821ad3" containerName="istio-proxy" Oct 30 18:37:27.296482 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.296461 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:27.318495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.318463 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:37:27.326706 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.326668 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9"] Oct 30 18:37:27.332241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.332206 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:37:27.332437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.332273 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:37:27.411986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.411912 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9\" (UID: \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:27.412229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.412205 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th4lz\" (UniqueName: \"kubernetes.io/projected/ad413fc7-2b98-4faf-af7d-285b1f4bc308-kube-api-access-th4lz\") pod \"maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9\" (UID: \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:27.412296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.412247 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9\" (UID: \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:27.419477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.419429 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:37:27.513259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.513227 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9\" (UID: \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:27.513768 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.513412 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4lz\" (UniqueName: \"kubernetes.io/projected/ad413fc7-2b98-4faf-af7d-285b1f4bc308-kube-api-access-th4lz\") pod \"maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9\" (UID: \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:27.513768 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.513444 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9\" (UID: \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:27.514076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.514051 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9\" (UID: \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:27.515930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.515906 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9\" (UID: \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:27.541825 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.541756 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th4lz\" (UniqueName: \"kubernetes.io/projected/ad413fc7-2b98-4faf-af7d-285b1f4bc308-kube-api-access-th4lz\") pod \"maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9\" (UID: \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:27.614593 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.614556 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:27.935385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:27.935189 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9"] Oct 30 18:37:27.938713 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:37:27.938679 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad413fc7_2b98_4faf_af7d_285b1f4bc308.slice/crio-f6c4c1c87a409f712546d7ab029d13ca5ec31e3aeb0e438883f3622a34913ee2 WatchSource:0}: Error finding container f6c4c1c87a409f712546d7ab029d13ca5ec31e3aeb0e438883f3622a34913ee2: Status 404 returned error can't find the container with id f6c4c1c87a409f712546d7ab029d13ca5ec31e3aeb0e438883f3622a34913ee2 Oct 30 18:37:28.761130 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:28.761084 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" event={"ID":"ad413fc7-2b98-4faf-af7d-285b1f4bc308","Type":"ContainerStarted","Data":"7097c5c4a35fe1ce266f58bdb52bc29d9345fc5e70586527192b777a9d8cbfdc"} Oct 30 18:37:28.762011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:28.761722 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" event={"ID":"ad413fc7-2b98-4faf-af7d-285b1f4bc308","Type":"ContainerStarted","Data":"f6c4c1c87a409f712546d7ab029d13ca5ec31e3aeb0e438883f3622a34913ee2"} Oct 30 18:37:28.813703 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:28.813637 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" podStartSLOduration=1.813618706 podStartE2EDuration="1.813618706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:28.811715842 +0000 UTC m=+2707.768719676" watchObservedRunningTime="2026-10-30 18:37:28.813618706 +0000 UTC m=+2707.770622541" Oct 30 18:37:29.769872 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:29.769836 4883 generic.go:356] "Generic (PLEG): container finished" podID="ad413fc7-2b98-4faf-af7d-285b1f4bc308" containerID="7097c5c4a35fe1ce266f58bdb52bc29d9345fc5e70586527192b777a9d8cbfdc" exitCode=0 Oct 30 18:37:29.770323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:29.769893 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" event={"ID":"ad413fc7-2b98-4faf-af7d-285b1f4bc308","Type":"ContainerDied","Data":"7097c5c4a35fe1ce266f58bdb52bc29d9345fc5e70586527192b777a9d8cbfdc"} Oct 30 18:37:29.993823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:29.993767 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq"] Oct 30 18:37:30.008983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:30.005183 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b00d69-zkzqq"] Oct 30 18:37:30.958569 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:30.958540 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:31.092812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.092782 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-xks-ca\") pod \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\" (UID: \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\") " Oct 30 18:37:31.092984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.092887 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-openshift-ca\") pod \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\" (UID: \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\") " Oct 30 18:37:31.093040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.092986 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad413fc7-2b98-4faf-af7d-285b1f4bc308-kube-api-access-th4lz\" (UniqueName: \"kubernetes.io/projected/ad413fc7-2b98-4faf-af7d-285b1f4bc308-kube-api-access-th4lz\") pod \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\" (UID: \"ad413fc7-2b98-4faf-af7d-285b1f4bc308\") " Oct 30 18:37:31.093310 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.093284 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-openshift-ca" pod "ad413fc7-2b98-4faf-af7d-285b1f4bc308" (UID: "ad413fc7-2b98-4faf-af7d-285b1f4bc308"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:31.093769 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.093748 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.095022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.094996 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-xks-ca" pod "ad413fc7-2b98-4faf-af7d-285b1f4bc308" (UID: "ad413fc7-2b98-4faf-af7d-285b1f4bc308"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:31.095096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.095080 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad413fc7-2b98-4faf-af7d-285b1f4bc308-kube-api-access-th4lz" pod "ad413fc7-2b98-4faf-af7d-285b1f4bc308" (UID: "ad413fc7-2b98-4faf-af7d-285b1f4bc308"). InnerVolumeSpecName "kube-api-access-th4lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:31.179213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.179179 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db" path="/var/lib/kubelet/pods/f8ee8d4d-c26b-4cae-b0ef-5f0e55b9e5db/volumes" Oct 30 18:37:31.195337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.195297 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad413fc7-2b98-4faf-af7d-285b1f4bc308-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.195337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.195329 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th4lz\" (UniqueName: \"kubernetes.io/projected/ad413fc7-2b98-4faf-af7d-285b1f4bc308-kube-api-access-th4lz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.699371 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.699298 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:37:31.785781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.785747 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" Oct 30 18:37:31.786003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.785774 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9" event={"ID":"ad413fc7-2b98-4faf-af7d-285b1f4bc308","Type":"ContainerDied","Data":"f6c4c1c87a409f712546d7ab029d13ca5ec31e3aeb0e438883f3622a34913ee2"} Oct 30 18:37:31.786003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.785862 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6c4c1c87a409f712546d7ab029d13ca5ec31e3aeb0e438883f3622a34913ee2" Oct 30 18:37:31.788332 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.788298 4883 generic.go:356] "Generic (PLEG): container finished" podID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerID="a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb" exitCode=0 Oct 30 18:37:31.788489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.788361 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" event={"ID":"63bb0b62-59e5-4aa2-b5d1-c8837731bf53","Type":"ContainerDied","Data":"a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb"} Oct 30 18:37:31.788489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.788432 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" event={"ID":"63bb0b62-59e5-4aa2-b5d1-c8837731bf53","Type":"ContainerDied","Data":"eed7a1dc338254a0cb0fae4b05819f2cf1babaad257f3f6ea51743b7517e90ac"} Oct 30 18:37:31.788489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.788464 4883 scope.go:122] "RemoveContainer" containerID="a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb" Oct 30 18:37:31.788655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.788641 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9" Oct 30 18:37:31.801480 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.801451 4883 scope.go:122] "RemoveContainer" containerID="40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181" Oct 30 18:37:31.809349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.809317 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-podinfo\") pod \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " Oct 30 18:37:31.809490 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.809360 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-credential-socket\") pod \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " Oct 30 18:37:31.809572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.809555 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istiod-ca-cert\") pod \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " Oct 30 18:37:31.809641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.809590 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-socket\") pod \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " Oct 30 18:37:31.809641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.809617 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-envoy\") pod \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " Oct 30 18:37:31.809762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.809702 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-certs\") pod \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " Oct 30 18:37:31.809762 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.809739 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-credential-socket" pod "63bb0b62-59e5-4aa2-b5d1-c8837731bf53" (UID: "63bb0b62-59e5-4aa2-b5d1-c8837731bf53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:31.809900 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.809881 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-kube-api-access-hwjq9\" (UniqueName: \"kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-kube-api-access-hwjq9\") pod \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " Oct 30 18:37:31.809992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.809915 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-token\" (UniqueName: \"kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-token\") pod \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " Oct 30 18:37:31.809992 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.809978 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-data\") pod \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\" (UID: \"63bb0b62-59e5-4aa2-b5d1-c8837731bf53\") " Oct 30 18:37:31.810150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.810110 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-certs" pod "63bb0b62-59e5-4aa2-b5d1-c8837731bf53" (UID: "63bb0b62-59e5-4aa2-b5d1-c8837731bf53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:31.810237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.810215 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-socket" pod "63bb0b62-59e5-4aa2-b5d1-c8837731bf53" (UID: "63bb0b62-59e5-4aa2-b5d1-c8837731bf53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:31.810452 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.810431 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-data" pod "63bb0b62-59e5-4aa2-b5d1-c8837731bf53" (UID: "63bb0b62-59e5-4aa2-b5d1-c8837731bf53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:31.810740 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.810715 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.810827 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.810781 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.810827 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.810798 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.810827 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.810811 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.811046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.810903 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istiod-ca-cert" pod "63bb0b62-59e5-4aa2-b5d1-c8837731bf53" (UID: "63bb0b62-59e5-4aa2-b5d1-c8837731bf53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:31.811745 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.811716 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-podinfo" pod "63bb0b62-59e5-4aa2-b5d1-c8837731bf53" (UID: "63bb0b62-59e5-4aa2-b5d1-c8837731bf53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:31.812718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.812691 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-token" pod "63bb0b62-59e5-4aa2-b5d1-c8837731bf53" (UID: "63bb0b62-59e5-4aa2-b5d1-c8837731bf53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:31.812841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.812822 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-envoy" pod "63bb0b62-59e5-4aa2-b5d1-c8837731bf53" (UID: "63bb0b62-59e5-4aa2-b5d1-c8837731bf53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:31.812905 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.812852 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-kube-api-access-hwjq9" pod "63bb0b62-59e5-4aa2-b5d1-c8837731bf53" (UID: "63bb0b62-59e5-4aa2-b5d1-c8837731bf53"). InnerVolumeSpecName "kube-api-access-hwjq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:31.813134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.813112 4883 scope.go:122] "RemoveContainer" containerID="a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb" Oct 30 18:37:31.813433 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:37:31.813412 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb\": container with ID starting with a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb not found: ID does not exist" containerID="a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb" Oct 30 18:37:31.813522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.813445 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb"} err="failed to get container status \"a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb\": rpc error: code = NotFound desc = could not find container \"a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb\": container with ID starting with a7ef77f4a167864573cd71c3b65fe008a6e74df065c94f06d437de5a0ba382fb not found: ID does not exist" Oct 30 18:37:31.813522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.813465 4883 scope.go:122] "RemoveContainer" containerID="40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181" Oct 30 18:37:31.813741 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:37:31.813721 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181\": container with ID starting with 40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181 not found: ID does not exist" containerID="40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181" Oct 30 18:37:31.813801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.813750 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181"} err="failed to get container status \"40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181\": rpc error: code = NotFound desc = could not find container \"40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181\": container with ID starting with 40d456119b15be38eb0c674c06a4c32ee1357f4f716e12b95c4521901e416181 not found: ID does not exist" Oct 30 18:37:31.912149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.912104 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwjq9\" (UniqueName: \"kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-kube-api-access-hwjq9\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.912149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.912144 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.912149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.912154 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.912149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.912162 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.912535 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.912171 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63bb0b62-59e5-4aa2-b5d1-c8837731bf53-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:31.957034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.956946 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl"] Oct 30 18:37:31.963881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.963853 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5"] Oct 30 18:37:31.986632 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:31.986603 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp"] Oct 30 18:37:32.236162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:32.236019 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9"] Oct 30 18:37:32.243151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:32.243101 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5cb3d5-openshift-default-d8bc49c97-nlwh9"] Oct 30 18:37:33.179524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:33.179493 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" path="/var/lib/kubelet/pods/63bb0b62-59e5-4aa2-b5d1-c8837731bf53/volumes" Oct 30 18:37:35.078702 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:37:35.078665 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:37:46.978375 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:46.978318 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" containerID="cri-o://baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42" gracePeriod=45 Oct 30 18:37:46.987382 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:46.987333 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" containerID="cri-o://dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1" gracePeriod=45 Oct 30 18:37:47.007232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.007198 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" containerID="cri-o://3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef" gracePeriod=45 Oct 30 18:37:47.069587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.069545 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": dial tcp 10.132.0.75:8000: connect: connection refused" Oct 30 18:37:47.069802 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.069778 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused" Oct 30 18:37:47.336577 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.336548 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:37:47.340578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.340405 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:37:47.345193 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.345163 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:37:47.415242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.415185 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-home\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-home\") pod \"5d215fd2-c798-4224-bff8-4d19fa58b773\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " Oct 30 18:37:47.415520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.415499 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-model-cache\") pod \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " Oct 30 18:37:47.415630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.415610 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-home" pod "5d215fd2-c798-4224-bff8-4d19fa58b773" (UID: "5d215fd2-c798-4224-bff8-4d19fa58b773"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.415726 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.415709 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-model-cache\") pod \"50f45f5e-25c0-4132-88e1-92b762f1d334\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " Oct 30 18:37:47.415833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.415772 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-model-cache\") pod \"5d215fd2-c798-4224-bff8-4d19fa58b773\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " Oct 30 18:37:47.415833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.415792 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-model-cache" pod "f930967b-8f0f-48d5-921a-0d1c7b1bf289" (UID: "f930967b-8f0f-48d5-921a-0d1c7b1bf289"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.415833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.415816 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-dshm\") pod \"50f45f5e-25c0-4132-88e1-92b762f1d334\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " Oct 30 18:37:47.416026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.415842 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-dshm\") pod \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " Oct 30 18:37:47.416026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.415925 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f930967b-8f0f-48d5-921a-0d1c7b1bf289-kube-api-access-b2vtp\" (UniqueName: \"kubernetes.io/projected/f930967b-8f0f-48d5-921a-0d1c7b1bf289-kube-api-access-b2vtp\") pod \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " Oct 30 18:37:47.416026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.415972 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d215fd2-c798-4224-bff8-4d19fa58b773-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d215fd2-c798-4224-bff8-4d19fa58b773-tls-certs\") pod \"5d215fd2-c798-4224-bff8-4d19fa58b773\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " Oct 30 18:37:47.416026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.415988 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-model-cache" pod "5d215fd2-c798-4224-bff8-4d19fa58b773" (UID: "5d215fd2-c798-4224-bff8-4d19fa58b773"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.416241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.416043 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-dshm\") pod \"5d215fd2-c798-4224-bff8-4d19fa58b773\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " Oct 30 18:37:47.416241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.416081 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-tmp-dir\") pod \"50f45f5e-25c0-4132-88e1-92b762f1d334\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " Oct 30 18:37:47.416241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.416111 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tls-certs\" (UniqueName: \"kubernetes.io/secret/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tls-certs\") pod \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " Oct 30 18:37:47.416241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.416150 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tmp-dir\") pod \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " Oct 30 18:37:47.416241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.416188 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-tmp-dir\") pod \"5d215fd2-c798-4224-bff8-4d19fa58b773\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " Oct 30 18:37:47.416241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.416215 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-home\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-home\") pod \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\" (UID: \"f930967b-8f0f-48d5-921a-0d1c7b1bf289\") " Oct 30 18:37:47.416718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.416078 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-model-cache" pod "50f45f5e-25c0-4132-88e1-92b762f1d334" (UID: "50f45f5e-25c0-4132-88e1-92b762f1d334"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.416802 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.416449 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-tmp-dir" pod "50f45f5e-25c0-4132-88e1-92b762f1d334" (UID: "50f45f5e-25c0-4132-88e1-92b762f1d334"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.416972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.416895 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tmp-dir" pod "f930967b-8f0f-48d5-921a-0d1c7b1bf289" (UID: "f930967b-8f0f-48d5-921a-0d1c7b1bf289"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.417466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.417053 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-tmp-dir" pod "5d215fd2-c798-4224-bff8-4d19fa58b773" (UID: "5d215fd2-c798-4224-bff8-4d19fa58b773"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.417466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.417303 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-home" pod "f930967b-8f0f-48d5-921a-0d1c7b1bf289" (UID: "f930967b-8f0f-48d5-921a-0d1c7b1bf289"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.417985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.417924 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d215fd2-c798-4224-bff8-4d19fa58b773-kube-api-access-s69hn\" (UniqueName: \"kubernetes.io/projected/5d215fd2-c798-4224-bff8-4d19fa58b773-kube-api-access-s69hn\") pod \"5d215fd2-c798-4224-bff8-4d19fa58b773\" (UID: \"5d215fd2-c798-4224-bff8-4d19fa58b773\") " Oct 30 18:37:47.418140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.418123 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f45f5e-25c0-4132-88e1-92b762f1d334-kube-api-access-btwg6\" (UniqueName: \"kubernetes.io/projected/50f45f5e-25c0-4132-88e1-92b762f1d334-kube-api-access-btwg6\") pod \"50f45f5e-25c0-4132-88e1-92b762f1d334\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " Oct 30 18:37:47.418379 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.418330 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-home\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-home\") pod \"50f45f5e-25c0-4132-88e1-92b762f1d334\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " Oct 30 18:37:47.418663 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.418647 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f45f5e-25c0-4132-88e1-92b762f1d334-tls-certs\" (UniqueName: \"kubernetes.io/secret/50f45f5e-25c0-4132-88e1-92b762f1d334-tls-certs\") pod \"50f45f5e-25c0-4132-88e1-92b762f1d334\" (UID: \"50f45f5e-25c0-4132-88e1-92b762f1d334\") " Oct 30 18:37:47.419066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.419041 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f930967b-8f0f-48d5-921a-0d1c7b1bf289-kube-api-access-b2vtp" pod "f930967b-8f0f-48d5-921a-0d1c7b1bf289" (UID: "f930967b-8f0f-48d5-921a-0d1c7b1bf289"). InnerVolumeSpecName "kube-api-access-b2vtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:47.419712 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.419692 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.419834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.419819 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.419925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.419913 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.422307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.421093 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f45f5e-25c0-4132-88e1-92b762f1d334-kube-api-access-btwg6" pod "50f45f5e-25c0-4132-88e1-92b762f1d334" (UID: "50f45f5e-25c0-4132-88e1-92b762f1d334"). InnerVolumeSpecName "kube-api-access-btwg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:47.422307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.421371 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-home" pod "50f45f5e-25c0-4132-88e1-92b762f1d334" (UID: "50f45f5e-25c0-4132-88e1-92b762f1d334"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.422533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.422516 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.422634 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.422620 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2vtp\" (UniqueName: \"kubernetes.io/projected/f930967b-8f0f-48d5-921a-0d1c7b1bf289-kube-api-access-b2vtp\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.422731 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.422718 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.422817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.422806 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.422912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.422900 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.423053 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.423039 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.424151 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.424131 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f45f5e-25c0-4132-88e1-92b762f1d334-tls-certs" pod "50f45f5e-25c0-4132-88e1-92b762f1d334" (UID: "50f45f5e-25c0-4132-88e1-92b762f1d334"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:47.425999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.425762 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tls-certs" pod "f930967b-8f0f-48d5-921a-0d1c7b1bf289" (UID: "f930967b-8f0f-48d5-921a-0d1c7b1bf289"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:47.425999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.425860 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-dshm" pod "5d215fd2-c798-4224-bff8-4d19fa58b773" (UID: "5d215fd2-c798-4224-bff8-4d19fa58b773"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.425999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.425971 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d215fd2-c798-4224-bff8-4d19fa58b773-tls-certs" pod "5d215fd2-c798-4224-bff8-4d19fa58b773" (UID: "5d215fd2-c798-4224-bff8-4d19fa58b773"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:47.426513 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.426488 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-dshm" pod "f930967b-8f0f-48d5-921a-0d1c7b1bf289" (UID: "f930967b-8f0f-48d5-921a-0d1c7b1bf289"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.430122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.430100 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-dshm" pod "50f45f5e-25c0-4132-88e1-92b762f1d334" (UID: "50f45f5e-25c0-4132-88e1-92b762f1d334"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:47.430360 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.430338 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d215fd2-c798-4224-bff8-4d19fa58b773-kube-api-access-s69hn" pod "5d215fd2-c798-4224-bff8-4d19fa58b773" (UID: "5d215fd2-c798-4224-bff8-4d19fa58b773"). InnerVolumeSpecName "kube-api-access-s69hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:47.525887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.524685 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.525887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.524715 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f930967b-8f0f-48d5-921a-0d1c7b1bf289-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.525887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.524727 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d215fd2-c798-4224-bff8-4d19fa58b773-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.525887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.524741 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d215fd2-c798-4224-bff8-4d19fa58b773-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.525887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.524753 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f930967b-8f0f-48d5-921a-0d1c7b1bf289-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.525887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.524768 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s69hn\" (UniqueName: \"kubernetes.io/projected/5d215fd2-c798-4224-bff8-4d19fa58b773-kube-api-access-s69hn\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.525887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.524783 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btwg6\" (UniqueName: \"kubernetes.io/projected/50f45f5e-25c0-4132-88e1-92b762f1d334-kube-api-access-btwg6\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.525887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.524796 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50f45f5e-25c0-4132-88e1-92b762f1d334-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:47.525887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:47.524807 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f45f5e-25c0-4132-88e1-92b762f1d334-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:48.030419 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.030384 4883 generic.go:356] "Generic (PLEG): container finished" podID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerID="baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42" exitCode=0 Oct 30 18:37:48.030845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.030424 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" Oct 30 18:37:48.030845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.030459 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" event={"ID":"f930967b-8f0f-48d5-921a-0d1c7b1bf289","Type":"ContainerDied","Data":"baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42"} Oct 30 18:37:48.030845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.030505 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl" event={"ID":"f930967b-8f0f-48d5-921a-0d1c7b1bf289","Type":"ContainerDied","Data":"eb133a1f311ac6152d0c4b7dc72de5232de003dc9a9f67f2f20beee10982a623"} Oct 30 18:37:48.030845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.030525 4883 scope.go:122] "RemoveContainer" containerID="baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42" Oct 30 18:37:48.032066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.032045 4883 generic.go:356] "Generic (PLEG): container finished" podID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerID="dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1" exitCode=0 Oct 30 18:37:48.032162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.032093 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" Oct 30 18:37:48.032162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.032150 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" event={"ID":"50f45f5e-25c0-4132-88e1-92b762f1d334","Type":"ContainerDied","Data":"dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1"} Oct 30 18:37:48.032285 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.032186 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5" event={"ID":"50f45f5e-25c0-4132-88e1-92b762f1d334","Type":"ContainerDied","Data":"b00e3a54fac1aed7639c22216637fd82ba66b44a16f3dc94cd7c6f5d789c7a56"} Oct 30 18:37:48.033963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.033907 4883 generic.go:356] "Generic (PLEG): container finished" podID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerID="3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef" exitCode=0 Oct 30 18:37:48.034082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.033971 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" event={"ID":"5d215fd2-c798-4224-bff8-4d19fa58b773","Type":"ContainerDied","Data":"3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef"} Oct 30 18:37:48.034082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.033989 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" Oct 30 18:37:48.034082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.033992 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp" event={"ID":"5d215fd2-c798-4224-bff8-4d19fa58b773","Type":"ContainerDied","Data":"e0aae7c9567e2dc88499359da96b150f0591b7f3a18f9df5a3755b43f54aad99"} Oct 30 18:37:48.043364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.043342 4883 scope.go:122] "RemoveContainer" containerID="baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42" Oct 30 18:37:48.043717 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:37:48.043686 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42\": container with ID starting with baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42 not found: ID does not exist" containerID="baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42" Oct 30 18:37:48.043894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.043725 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42"} err="failed to get container status \"baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42\": rpc error: code = NotFound desc = could not find container \"baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42\": container with ID starting with baec6116629a8460dd1264675fa9d4f09a3f0869f5fabbf79addb2fd82d8aa42 not found: ID does not exist" Oct 30 18:37:48.043894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.043748 4883 scope.go:122] "RemoveContainer" containerID="dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1" Oct 30 18:37:48.056369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.056351 4883 scope.go:122] "RemoveContainer" containerID="dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1" Oct 30 18:37:48.056661 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:37:48.056643 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1\": container with ID starting with dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1 not found: ID does not exist" containerID="dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1" Oct 30 18:37:48.056720 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.056673 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1"} err="failed to get container status \"dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1\": rpc error: code = NotFound desc = could not find container \"dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1\": container with ID starting with dc4a9f3508548b6aabf3242f8cdb3d226bfa0f837b06f11200d2bab06512fde1 not found: ID does not exist" Oct 30 18:37:48.056720 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.056692 4883 scope.go:122] "RemoveContainer" containerID="3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef" Oct 30 18:37:48.066039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.066013 4883 scope.go:122] "RemoveContainer" containerID="3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef" Oct 30 18:37:48.066325 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:37:48.066303 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef\": container with ID starting with 3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef not found: ID does not exist" containerID="3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef" Oct 30 18:37:48.066381 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.066335 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef"} err="failed to get container status \"3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef\": rpc error: code = NotFound desc = could not find container \"3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef\": container with ID starting with 3716d10b9678151266e66ac3eb9020a7dc46919c3c7e9dce3a286e705a95c1ef not found: ID does not exist" Oct 30 18:37:48.197980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.194985 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5"] Oct 30 18:37:48.201354 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.201325 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/distinct-w0-57ee15-kserve-545d7bc79c-jn6l5"] Oct 30 18:37:48.248752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.248661 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp"] Oct 30 18:37:48.263070 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.263040 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/simulator-w0-57ee15-kserve-dc9fb9dc5-g2rkp"] Oct 30 18:37:48.309631 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.309598 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl"] Oct 30 18:37:48.320700 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.320674 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-57ee15/distinct-2-w0-57ee15-kserve-7bdd66bf66-g9zzl"] Oct 30 18:37:48.984526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.982921 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf"] Oct 30 18:37:48.984526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.983443 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" podUID="081fffe0-7e1f-4562-b4c8-d8cb29c045bb" containerName="istio-proxy" containerID="cri-o://80df7a1577536df94586f4a26103032a034811ff425e5e7f0edb3db9b99a994e" gracePeriod=30 Oct 30 18:37:48.990574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.990547 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87"] Oct 30 18:37:48.990800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:48.990774 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" podUID="c816bfc8-985e-4cf5-865c-5adee66fb87d" containerName="istio-proxy" containerID="cri-o://77895149bea2ab91195cffdcb7fc8e824a3e2c67a7ea5d137350255d51baece4" gracePeriod=30 Oct 30 18:37:49.001040 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.001011 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf"] Oct 30 18:37:49.001220 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.001199 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" podUID="b558af39-00d4-41f8-b004-29437a7d270a" containerName="istio-proxy" containerID="cri-o://99a2827c7204af3f2daa9476d87a66a71cdb5e1f3eed3f5773fa0b736527231e" gracePeriod=30 Oct 30 18:37:49.232599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.232558 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" path="/var/lib/kubelet/pods/50f45f5e-25c0-4132-88e1-92b762f1d334/volumes" Oct 30 18:37:49.241350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.241278 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" path="/var/lib/kubelet/pods/5d215fd2-c798-4224-bff8-4d19fa58b773/volumes" Oct 30 18:37:49.253739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.253704 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" path="/var/lib/kubelet/pods/f930967b-8f0f-48d5-921a-0d1c7b1bf289/volumes" Oct 30 18:37:49.309150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.304416 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:37:49.309150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.304492 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" podUID="c816bfc8-985e-4cf5-865c-5adee66fb87d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:37:49.640606 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.640573 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:37:49.640783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.640645 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" podUID="081fffe0-7e1f-4562-b4c8-d8cb29c045bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:37:49.814852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.814821 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj"] Oct 30 18:37:49.815415 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815400 4883 cpu_manager.go:414] "containerMap: removing container" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" Oct 30 18:37:49.815465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815416 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" Oct 30 18:37:49.815465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815433 4883 cpu_manager.go:414] "containerMap: removing container" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" Oct 30 18:37:49.815465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815438 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" Oct 30 18:37:49.815465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815451 4883 cpu_manager.go:414] "containerMap: removing container" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerName="istio-proxy" Oct 30 18:37:49.815465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815456 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerName="istio-proxy" Oct 30 18:37:49.815638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815469 4883 cpu_manager.go:414] "containerMap: removing container" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" Oct 30 18:37:49.815638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815474 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" Oct 30 18:37:49.815638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815482 4883 cpu_manager.go:414] "containerMap: removing container" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerName="istio-proxy" Oct 30 18:37:49.815638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815486 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerName="istio-proxy" Oct 30 18:37:49.815638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815510 4883 cpu_manager.go:414] "containerMap: removing container" podUID="ad413fc7-2b98-4faf-af7d-285b1f4bc308" containerName="revoke-keys" Oct 30 18:37:49.815638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815515 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad413fc7-2b98-4faf-af7d-285b1f4bc308" containerName="revoke-keys" Oct 30 18:37:49.815638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815596 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad413fc7-2b98-4faf-af7d-285b1f4bc308" containerName="revoke-keys" Oct 30 18:37:49.815638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815607 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="f930967b-8f0f-48d5-921a-0d1c7b1bf289" containerName="main" Oct 30 18:37:49.815638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815613 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f45f5e-25c0-4132-88e1-92b762f1d334" containerName="main" Oct 30 18:37:49.815638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815620 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerName="istio-proxy" Oct 30 18:37:49.815638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815627 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d215fd2-c798-4224-bff8-4d19fa58b773" containerName="main" Oct 30 18:37:49.815638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.815631 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="63bb0b62-59e5-4aa2-b5d1-c8837731bf53" containerName="istio-proxy" Oct 30 18:37:49.820718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.820696 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:49.843185 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.843154 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:37:49.843354 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.843272 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:37:49.844073 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.844049 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj"] Oct 30 18:37:49.869702 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.869668 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj\" (UID: \"9c51dbdd-fc11-42b7-8918-3b957502118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:49.870072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.869964 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54gvd\" (UniqueName: \"kubernetes.io/projected/9c51dbdd-fc11-42b7-8918-3b957502118c-kube-api-access-54gvd\") pod \"maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj\" (UID: \"9c51dbdd-fc11-42b7-8918-3b957502118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:49.870072 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.870038 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj\" (UID: \"9c51dbdd-fc11-42b7-8918-3b957502118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:49.882208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.882177 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:37:49.882208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.882190 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:37:49.970998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.970876 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj\" (UID: \"9c51dbdd-fc11-42b7-8918-3b957502118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:49.971186 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.971169 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54gvd\" (UniqueName: \"kubernetes.io/projected/9c51dbdd-fc11-42b7-8918-3b957502118c-kube-api-access-54gvd\") pod \"maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj\" (UID: \"9c51dbdd-fc11-42b7-8918-3b957502118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:49.971272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.971232 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj\" (UID: \"9c51dbdd-fc11-42b7-8918-3b957502118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:49.971881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.971859 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj\" (UID: \"9c51dbdd-fc11-42b7-8918-3b957502118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:49.973416 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.973393 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj\" (UID: \"9c51dbdd-fc11-42b7-8918-3b957502118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:49.995583 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:49.995541 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54gvd\" (UniqueName: \"kubernetes.io/projected/9c51dbdd-fc11-42b7-8918-3b957502118c-kube-api-access-54gvd\") pod \"maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj\" (UID: \"9c51dbdd-fc11-42b7-8918-3b957502118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:50.136173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.136138 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:50.199718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.199687 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:37:50.199818 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.199748 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" podUID="b558af39-00d4-41f8-b004-29437a7d270a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:37:50.485721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.485693 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj"] Oct 30 18:37:50.488014 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:37:50.487971 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c51dbdd_fc11_42b7_8918_3b957502118c.slice/crio-98795ec2d84ac6f788572321f91a4a1a45549f9a0005173db99a1307d1ba34c9 WatchSource:0}: Error finding container 98795ec2d84ac6f788572321f91a4a1a45549f9a0005173db99a1307d1ba34c9: Status 404 returned error can't find the container with id 98795ec2d84ac6f788572321f91a4a1a45549f9a0005173db99a1307d1ba34c9 Oct 30 18:37:50.770142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.770051 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j"] Oct 30 18:37:50.776687 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.776591 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.795543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.795515 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-58d08b\"/\"default-dockercfg-gmm8s\"" Oct 30 18:37:50.795661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.795571 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-58d08b\"/\"autores-model-58d08b-kserve-self-signed-certs\"" Oct 30 18:37:50.803866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.803837 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j"] Oct 30 18:37:50.809393 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.809159 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-58d08b\"/\"kube-root-ca.crt\"" Oct 30 18:37:50.809393 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.809161 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-58d08b\"/\"openshift-service-ca.crt\"" Oct 30 18:37:50.883683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.883651 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-home\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.883824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.883770 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jftts\" (UniqueName: \"kubernetes.io/projected/df3fde75-25a2-4fde-b7a4-78e412c6c696-kube-api-access-jftts\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.883865 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.883824 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-tmp-dir\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.883865 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.883856 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3fde75-25a2-4fde-b7a4-78e412c6c696-tls-certs\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.884092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.884072 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-dshm\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.884137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.884107 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-model-cache\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.986035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.985998 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-dshm\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.986035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.986039 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-model-cache\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.986281 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.986214 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-home\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.986350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.986278 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftts\" (UniqueName: \"kubernetes.io/projected/df3fde75-25a2-4fde-b7a4-78e412c6c696-kube-api-access-jftts\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.986350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.986330 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-tmp-dir\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.986464 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.986365 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3fde75-25a2-4fde-b7a4-78e412c6c696-tls-certs\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.986582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.986546 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-model-cache\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.986660 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.986586 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-home\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.986867 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.986840 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-tmp-dir\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.988830 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.988803 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3fde75-25a2-4fde-b7a4-78e412c6c696-tls-certs\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:50.988999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:50.988977 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-dshm\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:51.014384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:51.014357 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jftts\" (UniqueName: \"kubernetes.io/projected/df3fde75-25a2-4fde-b7a4-78e412c6c696-kube-api-access-jftts\") pod \"autores-model-58d08b-kserve-7768d857f7-fln5j\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:51.133549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:51.132746 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:51.135331 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:51.135292 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" event={"ID":"9c51dbdd-fc11-42b7-8918-3b957502118c","Type":"ContainerStarted","Data":"7d33abb86324ba0aaa007fe1c2a012bb7f6de2ac01acbb9db85eb5d9ec266880"} Oct 30 18:37:51.135473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:51.135362 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" event={"ID":"9c51dbdd-fc11-42b7-8918-3b957502118c","Type":"ContainerStarted","Data":"98795ec2d84ac6f788572321f91a4a1a45549f9a0005173db99a1307d1ba34c9"} Oct 30 18:37:51.231245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:51.231124 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" podStartSLOduration=2.231108775 podStartE2EDuration="2.231108775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:51.226397682 +0000 UTC m=+2730.183401544" watchObservedRunningTime="2026-10-30 18:37:51.231108775 +0000 UTC m=+2730.188112613" Oct 30 18:37:51.527570 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:51.527537 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j"] Oct 30 18:37:51.529291 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:37:51.529242 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf3fde75_25a2_4fde_b7a4_78e412c6c696.slice/crio-8e345750960b6c75ae308134b2ff422209dbcc2574414f7c44e067764f0e301e WatchSource:0}: Error finding container 8e345750960b6c75ae308134b2ff422209dbcc2574414f7c44e067764f0e301e: Status 404 returned error can't find the container with id 8e345750960b6c75ae308134b2ff422209dbcc2574414f7c44e067764f0e301e Oct 30 18:37:52.150559 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:52.150520 4883 generic.go:356] "Generic (PLEG): container finished" podID="9c51dbdd-fc11-42b7-8918-3b957502118c" containerID="7d33abb86324ba0aaa007fe1c2a012bb7f6de2ac01acbb9db85eb5d9ec266880" exitCode=0 Oct 30 18:37:52.150983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:52.150961 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" event={"ID":"9c51dbdd-fc11-42b7-8918-3b957502118c","Type":"ContainerDied","Data":"7d33abb86324ba0aaa007fe1c2a012bb7f6de2ac01acbb9db85eb5d9ec266880"} Oct 30 18:37:52.154169 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:52.154138 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" event={"ID":"df3fde75-25a2-4fde-b7a4-78e412c6c696","Type":"ContainerStarted","Data":"0a1f22cef49d5861ec564ee9cf925668fdf2320069e6c61d64faba4ef427a4f2"} Oct 30 18:37:52.154289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:52.154180 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" event={"ID":"df3fde75-25a2-4fde-b7a4-78e412c6c696","Type":"ContainerStarted","Data":"8e345750960b6c75ae308134b2ff422209dbcc2574414f7c44e067764f0e301e"} Oct 30 18:37:52.257190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:52.257079 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" podStartSLOduration=2.257065254 podStartE2EDuration="2.257065254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:52.252894376 +0000 UTC m=+2731.209898209" watchObservedRunningTime="2026-10-30 18:37:52.257065254 +0000 UTC m=+2731.214069088" Oct 30 18:37:53.133714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:53.133678 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:37:53.324947 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:53.324912 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:53.456115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:53.456013 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c51dbdd-fc11-42b7-8918-3b957502118c-kube-api-access-54gvd\" (UniqueName: \"kubernetes.io/projected/9c51dbdd-fc11-42b7-8918-3b957502118c-kube-api-access-54gvd\") pod \"9c51dbdd-fc11-42b7-8918-3b957502118c\" (UID: \"9c51dbdd-fc11-42b7-8918-3b957502118c\") " Oct 30 18:37:53.456115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:53.456101 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-xks-ca\") pod \"9c51dbdd-fc11-42b7-8918-3b957502118c\" (UID: \"9c51dbdd-fc11-42b7-8918-3b957502118c\") " Oct 30 18:37:53.456411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:53.456372 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-openshift-ca\") pod \"9c51dbdd-fc11-42b7-8918-3b957502118c\" (UID: \"9c51dbdd-fc11-42b7-8918-3b957502118c\") " Oct 30 18:37:53.457298 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:53.457264 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-openshift-ca" pod "9c51dbdd-fc11-42b7-8918-3b957502118c" (UID: "9c51dbdd-fc11-42b7-8918-3b957502118c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:53.458700 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:53.458671 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-xks-ca" pod "9c51dbdd-fc11-42b7-8918-3b957502118c" (UID: "9c51dbdd-fc11-42b7-8918-3b957502118c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:53.458793 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:53.458706 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c51dbdd-fc11-42b7-8918-3b957502118c-kube-api-access-54gvd" pod "9c51dbdd-fc11-42b7-8918-3b957502118c" (UID: "9c51dbdd-fc11-42b7-8918-3b957502118c"). InnerVolumeSpecName "kube-api-access-54gvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:53.558572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:53.558541 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.558572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:53.558569 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54gvd\" (UniqueName: \"kubernetes.io/projected/9c51dbdd-fc11-42b7-8918-3b957502118c-kube-api-access-54gvd\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.558572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:53.558579 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c51dbdd-fc11-42b7-8918-3b957502118c-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.171247 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.171217 4883 generic.go:356] "Generic (PLEG): container finished" podID="b558af39-00d4-41f8-b004-29437a7d270a" containerID="99a2827c7204af3f2daa9476d87a66a71cdb5e1f3eed3f5773fa0b736527231e" exitCode=0 Oct 30 18:37:54.171850 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.171719 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" event={"ID":"b558af39-00d4-41f8-b004-29437a7d270a","Type":"ContainerDied","Data":"99a2827c7204af3f2daa9476d87a66a71cdb5e1f3eed3f5773fa0b736527231e"} Oct 30 18:37:54.192550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.173255 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" event={"ID":"9c51dbdd-fc11-42b7-8918-3b957502118c","Type":"ContainerDied","Data":"98795ec2d84ac6f788572321f91a4a1a45549f9a0005173db99a1307d1ba34c9"} Oct 30 18:37:54.192550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.173288 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj" Oct 30 18:37:54.192550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.173292 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98795ec2d84ac6f788572321f91a4a1a45549f9a0005173db99a1307d1ba34c9" Oct 30 18:37:54.192550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.174905 4883 generic.go:356] "Generic (PLEG): container finished" podID="c816bfc8-985e-4cf5-865c-5adee66fb87d" containerID="77895149bea2ab91195cffdcb7fc8e824a3e2c67a7ea5d137350255d51baece4" exitCode=0 Oct 30 18:37:54.192550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.175032 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" event={"ID":"c816bfc8-985e-4cf5-865c-5adee66fb87d","Type":"ContainerDied","Data":"77895149bea2ab91195cffdcb7fc8e824a3e2c67a7ea5d137350255d51baece4"} Oct 30 18:37:54.192550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.176590 4883 generic.go:356] "Generic (PLEG): container finished" podID="081fffe0-7e1f-4562-b4c8-d8cb29c045bb" containerID="80df7a1577536df94586f4a26103032a034811ff425e5e7f0edb3db9b99a994e" exitCode=0 Oct 30 18:37:54.192550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.176742 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" event={"ID":"081fffe0-7e1f-4562-b4c8-d8cb29c045bb","Type":"ContainerDied","Data":"80df7a1577536df94586f4a26103032a034811ff425e5e7f0edb3db9b99a994e"} Oct 30 18:37:54.283537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.283505 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:37:54.369807 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.369778 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:37:54.370470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.370444 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-envoy\") pod \"c816bfc8-985e-4cf5-865c-5adee66fb87d\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " Oct 30 18:37:54.370548 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.370525 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-data\") pod \"c816bfc8-985e-4cf5-865c-5adee66fb87d\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " Oct 30 18:37:54.370618 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.370595 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-kube-api-access-vwqrm\" (UniqueName: \"kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-kube-api-access-vwqrm\") pod \"c816bfc8-985e-4cf5-865c-5adee66fb87d\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " Oct 30 18:37:54.371337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.370909 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-certs\") pod \"c816bfc8-985e-4cf5-865c-5adee66fb87d\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " Oct 30 18:37:54.371337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.370974 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c816bfc8-985e-4cf5-865c-5adee66fb87d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c816bfc8-985e-4cf5-865c-5adee66fb87d-istiod-ca-cert\") pod \"c816bfc8-985e-4cf5-865c-5adee66fb87d\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " Oct 30 18:37:54.371337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.371254 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-data" pod "c816bfc8-985e-4cf5-865c-5adee66fb87d" (UID: "c816bfc8-985e-4cf5-865c-5adee66fb87d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.371337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.371297 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-certs" pod "c816bfc8-985e-4cf5-865c-5adee66fb87d" (UID: "c816bfc8-985e-4cf5-865c-5adee66fb87d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.371624 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.371503 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-credential-socket\") pod \"c816bfc8-985e-4cf5-865c-5adee66fb87d\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " Oct 30 18:37:54.371624 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.371537 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-token\" (UniqueName: \"kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-token\") pod \"c816bfc8-985e-4cf5-865c-5adee66fb87d\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " Oct 30 18:37:54.371730 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.371622 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-podinfo\") pod \"c816bfc8-985e-4cf5-865c-5adee66fb87d\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " Oct 30 18:37:54.371730 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.371663 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c816bfc8-985e-4cf5-865c-5adee66fb87d-istiod-ca-cert" pod "c816bfc8-985e-4cf5-865c-5adee66fb87d" (UID: "c816bfc8-985e-4cf5-865c-5adee66fb87d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:54.371730 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.371690 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-socket\") pod \"c816bfc8-985e-4cf5-865c-5adee66fb87d\" (UID: \"c816bfc8-985e-4cf5-865c-5adee66fb87d\") " Oct 30 18:37:54.372486 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.372455 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-socket" pod "c816bfc8-985e-4cf5-865c-5adee66fb87d" (UID: "c816bfc8-985e-4cf5-865c-5adee66fb87d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.372486 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.372472 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.372620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.372494 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.372620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.372508 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c816bfc8-985e-4cf5-865c-5adee66fb87d-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.372918 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.372885 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-credential-socket" pod "c816bfc8-985e-4cf5-865c-5adee66fb87d" (UID: "c816bfc8-985e-4cf5-865c-5adee66fb87d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.375060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.375033 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-envoy" pod "c816bfc8-985e-4cf5-865c-5adee66fb87d" (UID: "c816bfc8-985e-4cf5-865c-5adee66fb87d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.377514 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.377494 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-podinfo" pod "c816bfc8-985e-4cf5-865c-5adee66fb87d" (UID: "c816bfc8-985e-4cf5-865c-5adee66fb87d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:54.377586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.377523 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-token" pod "c816bfc8-985e-4cf5-865c-5adee66fb87d" (UID: "c816bfc8-985e-4cf5-865c-5adee66fb87d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.378095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.378076 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-kube-api-access-vwqrm" pod "c816bfc8-985e-4cf5-865c-5adee66fb87d" (UID: "c816bfc8-985e-4cf5-865c-5adee66fb87d"). InnerVolumeSpecName "kube-api-access-vwqrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.404180 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.404155 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:37:54.469363 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.469331 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb"] Oct 30 18:37:54.473836 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.473629 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-data\") pod \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " Oct 30 18:37:54.474060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.473968 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-data" pod "081fffe0-7e1f-4562-b4c8-d8cb29c045bb" (UID: "081fffe0-7e1f-4562-b4c8-d8cb29c045bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.474142 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.474086 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-certs\") pod \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " Oct 30 18:37:54.474207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.474157 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-credential-socket\") pod \"b558af39-00d4-41f8-b004-29437a7d270a\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " Oct 30 18:37:54.474890 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.474849 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-credential-socket\") pod \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " Oct 30 18:37:54.474890 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.474431 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-credential-socket" pod "b558af39-00d4-41f8-b004-29437a7d270a" (UID: "b558af39-00d4-41f8-b004-29437a7d270a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.475173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.474635 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-certs" pod "081fffe0-7e1f-4562-b4c8-d8cb29c045bb" (UID: "081fffe0-7e1f-4562-b4c8-d8cb29c045bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.475173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.474894 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-envoy\") pod \"b558af39-00d4-41f8-b004-29437a7d270a\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " Oct 30 18:37:54.475173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.474923 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-envoy\") pod \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " Oct 30 18:37:54.475173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475069 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-socket\") pod \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " Oct 30 18:37:54.475173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475139 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istiod-ca-cert\") pod \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " Oct 30 18:37:54.475520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475188 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-kube-api-access-7rxj4\" (UniqueName: \"kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-kube-api-access-7rxj4\") pod \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " Oct 30 18:37:54.475520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475231 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-certs\") pod \"b558af39-00d4-41f8-b004-29437a7d270a\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " Oct 30 18:37:54.475520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475254 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-credential-socket" pod "081fffe0-7e1f-4562-b4c8-d8cb29c045bb" (UID: "081fffe0-7e1f-4562-b4c8-d8cb29c045bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.475520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475295 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b558af39-00d4-41f8-b004-29437a7d270a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b558af39-00d4-41f8-b004-29437a7d270a-istio-podinfo\") pod \"b558af39-00d4-41f8-b004-29437a7d270a\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " Oct 30 18:37:54.475520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475333 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-token\" (UniqueName: \"kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-token\") pod \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " Oct 30 18:37:54.475520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475368 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b558af39-00d4-41f8-b004-29437a7d270a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b558af39-00d4-41f8-b004-29437a7d270a-istiod-ca-cert\") pod \"b558af39-00d4-41f8-b004-29437a7d270a\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " Oct 30 18:37:54.475520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475424 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-istio-token\" (UniqueName: \"kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-istio-token\") pod \"b558af39-00d4-41f8-b004-29437a7d270a\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " Oct 30 18:37:54.475520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475470 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-socket" pod "081fffe0-7e1f-4562-b4c8-d8cb29c045bb" (UID: "081fffe0-7e1f-4562-b4c8-d8cb29c045bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.475520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475488 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-podinfo\") pod \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\" (UID: \"081fffe0-7e1f-4562-b4c8-d8cb29c045bb\") " Oct 30 18:37:54.475971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475527 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-socket\") pod \"b558af39-00d4-41f8-b004-29437a7d270a\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " Oct 30 18:37:54.475971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475575 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-kube-api-access-8srnm\" (UniqueName: \"kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-kube-api-access-8srnm\") pod \"b558af39-00d4-41f8-b004-29437a7d270a\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " Oct 30 18:37:54.475971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.475603 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-data\") pod \"b558af39-00d4-41f8-b004-29437a7d270a\" (UID: \"b558af39-00d4-41f8-b004-29437a7d270a\") " Oct 30 18:37:54.477162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.477138 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b558af39-00d4-41f8-b004-29437a7d270a-istiod-ca-cert" pod "b558af39-00d4-41f8-b004-29437a7d270a" (UID: "b558af39-00d4-41f8-b004-29437a7d270a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:54.490400 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.477833 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.490419 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.490438 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwqrm\" (UniqueName: \"kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-kube-api-access-vwqrm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.490454 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.490466 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.490479 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.490492 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c816bfc8-985e-4cf5-865c-5adee66fb87d-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.490503 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c816bfc8-985e-4cf5-865c-5adee66fb87d-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.490517 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.490530 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.490542 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.478378 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-certs" pod "b558af39-00d4-41f8-b004-29437a7d270a" (UID: "b558af39-00d4-41f8-b004-29437a7d270a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.478483 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-socket" pod "b558af39-00d4-41f8-b004-29437a7d270a" (UID: "b558af39-00d4-41f8-b004-29437a7d270a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.490566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.478681 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istiod-ca-cert" pod "081fffe0-7e1f-4562-b4c8-d8cb29c045bb" (UID: "081fffe0-7e1f-4562-b4c8-d8cb29c045bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:54.491207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.479557 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-data" pod "b558af39-00d4-41f8-b004-29437a7d270a" (UID: "b558af39-00d4-41f8-b004-29437a7d270a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.491207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.482715 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b558af39-00d4-41f8-b004-29437a7d270a-istio-podinfo" pod "b558af39-00d4-41f8-b004-29437a7d270a" (UID: "b558af39-00d4-41f8-b004-29437a7d270a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:54.491207 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.485984 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp"] Oct 30 18:37:54.491506 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.491486 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-kube-api-access-7rxj4" pod "081fffe0-7e1f-4562-b4c8-d8cb29c045bb" (UID: "081fffe0-7e1f-4562-b4c8-d8cb29c045bb"). InnerVolumeSpecName "kube-api-access-7rxj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.493277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.493224 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-kube-api-access-8srnm" pod "b558af39-00d4-41f8-b004-29437a7d270a" (UID: "b558af39-00d4-41f8-b004-29437a7d270a"). InnerVolumeSpecName "kube-api-access-8srnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.498861 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.498835 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl"] Oct 30 18:37:54.499469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.499428 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-envoy" pod "b558af39-00d4-41f8-b004-29437a7d270a" (UID: "b558af39-00d4-41f8-b004-29437a7d270a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.499469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.499445 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-podinfo" pod "081fffe0-7e1f-4562-b4c8-d8cb29c045bb" (UID: "081fffe0-7e1f-4562-b4c8-d8cb29c045bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:54.499615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.499473 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-token" pod "081fffe0-7e1f-4562-b4c8-d8cb29c045bb" (UID: "081fffe0-7e1f-4562-b4c8-d8cb29c045bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.499778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.499746 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-envoy" pod "081fffe0-7e1f-4562-b4c8-d8cb29c045bb" (UID: "081fffe0-7e1f-4562-b4c8-d8cb29c045bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:54.501141 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.501115 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-istio-token" pod "b558af39-00d4-41f8-b004-29437a7d270a" (UID: "b558af39-00d4-41f8-b004-29437a7d270a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:54.591874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591790 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.591874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591825 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.591874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591837 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.591874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591850 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rxj4\" (UniqueName: \"kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-kube-api-access-7rxj4\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.591874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591863 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.591874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591875 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b558af39-00d4-41f8-b004-29437a7d270a-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.592226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591887 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.592226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591898 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b558af39-00d4-41f8-b004-29437a7d270a-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.592226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591910 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.592226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591920 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081fffe0-7e1f-4562-b4c8-d8cb29c045bb-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.592226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591945 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.592226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591959 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8srnm\" (UniqueName: \"kubernetes.io/projected/b558af39-00d4-41f8-b004-29437a7d270a-kube-api-access-8srnm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:54.592226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:54.591971 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b558af39-00d4-41f8-b004-29437a7d270a-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:37:55.201425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.201111 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" event={"ID":"081fffe0-7e1f-4562-b4c8-d8cb29c045bb","Type":"ContainerDied","Data":"59189e1d6a193fbb05af7d9e58948499882736f4c2b6522b1e436e99160da8e7"} Oct 30 18:37:55.201425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.201186 4883 scope.go:122] "RemoveContainer" containerID="80df7a1577536df94586f4a26103032a034811ff425e5e7f0edb3db9b99a994e" Oct 30 18:37:55.201425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.201374 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf" Oct 30 18:37:55.213308 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.213266 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" Oct 30 18:37:55.213566 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.213126 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf" event={"ID":"b558af39-00d4-41f8-b004-29437a7d270a","Type":"ContainerDied","Data":"098f8adad0c738130188f52402e544e5f4d78b6567bd748c017cb9cf2ea9db92"} Oct 30 18:37:55.215205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.215160 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" event={"ID":"c816bfc8-985e-4cf5-865c-5adee66fb87d","Type":"ContainerDied","Data":"f1f2f4048cd1e27859b51dafb5873cbdc546b97d729926426b1f84a6d724cf62"} Oct 30 18:37:55.215291 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.215234 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87" Oct 30 18:37:55.222817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.222797 4883 scope.go:122] "RemoveContainer" containerID="99a2827c7204af3f2daa9476d87a66a71cdb5e1f3eed3f5773fa0b736527231e" Oct 30 18:37:55.233019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.233000 4883 scope.go:122] "RemoveContainer" containerID="77895149bea2ab91195cffdcb7fc8e824a3e2c67a7ea5d137350255d51baece4" Oct 30 18:37:55.275027 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.275000 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf"] Oct 30 18:37:55.288848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.288821 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-cjwxf"] Oct 30 18:37:55.318653 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.318618 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87"] Oct 30 18:37:55.323246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.323215 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-kfj87"] Oct 30 18:37:55.397055 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.397017 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf"] Oct 30 18:37:55.402277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:55.402236 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b00d69-openshift-default-6556c878f5-jzmrf"] Oct 30 18:37:57.178515 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:57.178482 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081fffe0-7e1f-4562-b4c8-d8cb29c045bb" path="/var/lib/kubelet/pods/081fffe0-7e1f-4562-b4c8-d8cb29c045bb/volumes" Oct 30 18:37:57.179003 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:57.178988 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b558af39-00d4-41f8-b004-29437a7d270a" path="/var/lib/kubelet/pods/b558af39-00d4-41f8-b004-29437a7d270a/volumes" Oct 30 18:37:57.179427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:57.179414 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c816bfc8-985e-4cf5-865c-5adee66fb87d" path="/var/lib/kubelet/pods/c816bfc8-985e-4cf5-865c-5adee66fb87d/volumes" Oct 30 18:37:58.830374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.830342 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj"] Oct 30 18:37:58.831135 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.831119 4883 cpu_manager.go:414] "containerMap: removing container" podUID="b558af39-00d4-41f8-b004-29437a7d270a" containerName="istio-proxy" Oct 30 18:37:58.831210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.831136 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b558af39-00d4-41f8-b004-29437a7d270a" containerName="istio-proxy" Oct 30 18:37:58.831210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.831157 4883 cpu_manager.go:414] "containerMap: removing container" podUID="9c51dbdd-fc11-42b7-8918-3b957502118c" containerName="revoke-keys" Oct 30 18:37:58.831210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.831164 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c51dbdd-fc11-42b7-8918-3b957502118c" containerName="revoke-keys" Oct 30 18:37:58.831210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.831177 4883 cpu_manager.go:414] "containerMap: removing container" podUID="081fffe0-7e1f-4562-b4c8-d8cb29c045bb" containerName="istio-proxy" Oct 30 18:37:58.831210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.831182 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081fffe0-7e1f-4562-b4c8-d8cb29c045bb" containerName="istio-proxy" Oct 30 18:37:58.831210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.831193 4883 cpu_manager.go:414] "containerMap: removing container" podUID="c816bfc8-985e-4cf5-865c-5adee66fb87d" containerName="istio-proxy" Oct 30 18:37:58.831210 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.831198 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c816bfc8-985e-4cf5-865c-5adee66fb87d" containerName="istio-proxy" Oct 30 18:37:58.831432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.831303 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c51dbdd-fc11-42b7-8918-3b957502118c" containerName="revoke-keys" Oct 30 18:37:58.831432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.831315 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="081fffe0-7e1f-4562-b4c8-d8cb29c045bb" containerName="istio-proxy" Oct 30 18:37:58.831432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.831322 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="c816bfc8-985e-4cf5-865c-5adee66fb87d" containerName="istio-proxy" Oct 30 18:37:58.831432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.831335 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="b558af39-00d4-41f8-b004-29437a7d270a" containerName="istio-proxy" Oct 30 18:37:58.841288 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.841249 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:37:58.853517 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.853489 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r"] Oct 30 18:37:58.858085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.858067 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:37:58.884565 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.884537 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:37:58.885204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.884532 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:37:58.885426 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.885400 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:37:58.885504 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.885446 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:37:58.892993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.892921 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj"] Oct 30 18:37:58.898365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.898336 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r"] Oct 30 18:37:58.964642 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.964604 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-044a8b97-cjpfj\" (UID: \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:37:58.964871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.964808 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvjc7\" (UniqueName: \"kubernetes.io/projected/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-kube-api-access-wvjc7\") pod \"maas-api-revoke-sub-044a8b97-cjpfj\" (UID: \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:37:58.964871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.964847 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec1f92be-9mk4r\" (UID: \"64f9a074-8927-4f97-9287-6105c91e6bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:37:58.965025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.964974 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtdcv\" (UniqueName: \"kubernetes.io/projected/64f9a074-8927-4f97-9287-6105c91e6bfb-kube-api-access-vtdcv\") pod \"maas-api-revoke-sub-ec1f92be-9mk4r\" (UID: \"64f9a074-8927-4f97-9287-6105c91e6bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:37:58.965096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.965063 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec1f92be-9mk4r\" (UID: \"64f9a074-8927-4f97-9287-6105c91e6bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:37:58.965152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:58.965097 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-044a8b97-cjpfj\" (UID: \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:37:59.066290 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.066248 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-044a8b97-cjpfj\" (UID: \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:37:59.066469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.066395 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvjc7\" (UniqueName: \"kubernetes.io/projected/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-kube-api-access-wvjc7\") pod \"maas-api-revoke-sub-044a8b97-cjpfj\" (UID: \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:37:59.066469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.066424 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec1f92be-9mk4r\" (UID: \"64f9a074-8927-4f97-9287-6105c91e6bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:37:59.066469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.066448 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtdcv\" (UniqueName: \"kubernetes.io/projected/64f9a074-8927-4f97-9287-6105c91e6bfb-kube-api-access-vtdcv\") pod \"maas-api-revoke-sub-ec1f92be-9mk4r\" (UID: \"64f9a074-8927-4f97-9287-6105c91e6bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:37:59.066627 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.066516 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec1f92be-9mk4r\" (UID: \"64f9a074-8927-4f97-9287-6105c91e6bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:37:59.066627 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.066543 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-044a8b97-cjpfj\" (UID: \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:37:59.067057 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.067029 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-044a8b97-cjpfj\" (UID: \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:37:59.067206 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.067184 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec1f92be-9mk4r\" (UID: \"64f9a074-8927-4f97-9287-6105c91e6bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:37:59.069037 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.069015 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec1f92be-9mk4r\" (UID: \"64f9a074-8927-4f97-9287-6105c91e6bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:37:59.069137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.069066 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-044a8b97-cjpfj\" (UID: \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:37:59.091981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.091903 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtdcv\" (UniqueName: \"kubernetes.io/projected/64f9a074-8927-4f97-9287-6105c91e6bfb-kube-api-access-vtdcv\") pod \"maas-api-revoke-sub-ec1f92be-9mk4r\" (UID: \"64f9a074-8927-4f97-9287-6105c91e6bfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:37:59.092088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.091978 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvjc7\" (UniqueName: \"kubernetes.io/projected/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-kube-api-access-wvjc7\") pod \"maas-api-revoke-sub-044a8b97-cjpfj\" (UID: \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:37:59.159600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.159567 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:37:59.177855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.177832 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:37:59.476062 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.476033 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj"] Oct 30 18:37:59.477525 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:37:59.477498 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a3efb9_7e77_4ed8_a69b_25f7d1446b19.slice/crio-194652d88e5f84453d84c7a25ad1b72da280d203dce81d5ca1ad07f6acd5dd8b WatchSource:0}: Error finding container 194652d88e5f84453d84c7a25ad1b72da280d203dce81d5ca1ad07f6acd5dd8b: Status 404 returned error can't find the container with id 194652d88e5f84453d84c7a25ad1b72da280d203dce81d5ca1ad07f6acd5dd8b Oct 30 18:37:59.511801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:37:59.511774 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r"] Oct 30 18:37:59.513801 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:37:59.513776 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f9a074_8927_4f97_9287_6105c91e6bfb.slice/crio-3767c0da682db259d98af49db2af371b74158772a844d8c6e66d7975b1ffacf2 WatchSource:0}: Error finding container 3767c0da682db259d98af49db2af371b74158772a844d8c6e66d7975b1ffacf2: Status 404 returned error can't find the container with id 3767c0da682db259d98af49db2af371b74158772a844d8c6e66d7975b1ffacf2 Oct 30 18:38:00.350485 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:00.350438 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" event={"ID":"70a3efb9-7e77-4ed8-a69b-25f7d1446b19","Type":"ContainerStarted","Data":"93203434a0efffc2813b9c124046cbf574a40500ae6df48bc1bf98329c9143e9"} Oct 30 18:38:00.351177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:00.351138 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" event={"ID":"70a3efb9-7e77-4ed8-a69b-25f7d1446b19","Type":"ContainerStarted","Data":"194652d88e5f84453d84c7a25ad1b72da280d203dce81d5ca1ad07f6acd5dd8b"} Oct 30 18:38:00.357502 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:00.355580 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" event={"ID":"64f9a074-8927-4f97-9287-6105c91e6bfb","Type":"ContainerStarted","Data":"3dbf576228dee177478beb748ef557e5f0b9a0a79db8365efdc4b24ecd47ea1d"} Oct 30 18:38:00.357502 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:00.355638 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" event={"ID":"64f9a074-8927-4f97-9287-6105c91e6bfb","Type":"ContainerStarted","Data":"3767c0da682db259d98af49db2af371b74158772a844d8c6e66d7975b1ffacf2"} Oct 30 18:38:00.408152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:00.408096 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" podStartSLOduration=2.408080464 podStartE2EDuration="2.408080464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:00.406614696 +0000 UTC m=+2739.363618532" watchObservedRunningTime="2026-10-30 18:38:00.408080464 +0000 UTC m=+2739.365084299" Oct 30 18:38:00.459454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:00.459392 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" podStartSLOduration=2.45937759 podStartE2EDuration="2.45937759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:00.458572861 +0000 UTC m=+2739.415576698" watchObservedRunningTime="2026-10-30 18:38:00.45937759 +0000 UTC m=+2739.416381426" Oct 30 18:38:01.136958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:01.136637 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:38:01.154582 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:01.154549 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:38:01.402010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:01.398113 4883 generic.go:356] "Generic (PLEG): container finished" podID="70a3efb9-7e77-4ed8-a69b-25f7d1446b19" containerID="93203434a0efffc2813b9c124046cbf574a40500ae6df48bc1bf98329c9143e9" exitCode=0 Oct 30 18:38:01.402010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:01.398248 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" event={"ID":"70a3efb9-7e77-4ed8-a69b-25f7d1446b19","Type":"ContainerDied","Data":"93203434a0efffc2813b9c124046cbf574a40500ae6df48bc1bf98329c9143e9"} Oct 30 18:38:01.413970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:01.411950 4883 generic.go:356] "Generic (PLEG): container finished" podID="64f9a074-8927-4f97-9287-6105c91e6bfb" containerID="3dbf576228dee177478beb748ef557e5f0b9a0a79db8365efdc4b24ecd47ea1d" exitCode=0 Oct 30 18:38:01.413970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:01.413470 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" event={"ID":"64f9a074-8927-4f97-9287-6105c91e6bfb","Type":"ContainerDied","Data":"3dbf576228dee177478beb748ef557e5f0b9a0a79db8365efdc4b24ecd47ea1d"} Oct 30 18:38:01.718234 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:01.717897 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:38:02.748451 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.748419 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:38:02.752254 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.752227 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:38:02.919404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.919356 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-xks-ca\") pod \"64f9a074-8927-4f97-9287-6105c91e6bfb\" (UID: \"64f9a074-8927-4f97-9287-6105c91e6bfb\") " Oct 30 18:38:02.919619 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.919456 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-openshift-ca\") pod \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\" (UID: \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\") " Oct 30 18:38:02.919619 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.919565 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-xks-ca\") pod \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\" (UID: \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\") " Oct 30 18:38:02.919774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.919626 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-openshift-ca\") pod \"64f9a074-8927-4f97-9287-6105c91e6bfb\" (UID: \"64f9a074-8927-4f97-9287-6105c91e6bfb\") " Oct 30 18:38:02.919774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.919688 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-kube-api-access-wvjc7\" (UniqueName: \"kubernetes.io/projected/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-kube-api-access-wvjc7\") pod \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\" (UID: \"70a3efb9-7e77-4ed8-a69b-25f7d1446b19\") " Oct 30 18:38:02.919774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.919742 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f9a074-8927-4f97-9287-6105c91e6bfb-kube-api-access-vtdcv\" (UniqueName: \"kubernetes.io/projected/64f9a074-8927-4f97-9287-6105c91e6bfb-kube-api-access-vtdcv\") pod \"64f9a074-8927-4f97-9287-6105c91e6bfb\" (UID: \"64f9a074-8927-4f97-9287-6105c91e6bfb\") " Oct 30 18:38:02.922195 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.922158 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-openshift-ca" pod "64f9a074-8927-4f97-9287-6105c91e6bfb" (UID: "64f9a074-8927-4f97-9287-6105c91e6bfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:02.922351 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.922327 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-openshift-ca" pod "70a3efb9-7e77-4ed8-a69b-25f7d1446b19" (UID: "70a3efb9-7e77-4ed8-a69b-25f7d1446b19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:02.926277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.926235 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-kube-api-access-wvjc7" pod "70a3efb9-7e77-4ed8-a69b-25f7d1446b19" (UID: "70a3efb9-7e77-4ed8-a69b-25f7d1446b19"). InnerVolumeSpecName "kube-api-access-wvjc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:02.930587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.930549 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-xks-ca" pod "64f9a074-8927-4f97-9287-6105c91e6bfb" (UID: "64f9a074-8927-4f97-9287-6105c91e6bfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:02.937243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.937221 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f9a074-8927-4f97-9287-6105c91e6bfb-kube-api-access-vtdcv" pod "64f9a074-8927-4f97-9287-6105c91e6bfb" (UID: "64f9a074-8927-4f97-9287-6105c91e6bfb"). InnerVolumeSpecName "kube-api-access-vtdcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:02.938606 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:02.938443 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-xks-ca" pod "70a3efb9-7e77-4ed8-a69b-25f7d1446b19" (UID: "70a3efb9-7e77-4ed8-a69b-25f7d1446b19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:03.046097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:03.025147 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:03.046097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:03.025184 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:03.046097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:03.025201 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvjc7\" (UniqueName: \"kubernetes.io/projected/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-kube-api-access-wvjc7\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:03.046097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:03.025218 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtdcv\" (UniqueName: \"kubernetes.io/projected/64f9a074-8927-4f97-9287-6105c91e6bfb-kube-api-access-vtdcv\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:03.046097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:03.025232 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f9a074-8927-4f97-9287-6105c91e6bfb-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:03.046097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:03.025246 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70a3efb9-7e77-4ed8-a69b-25f7d1446b19-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:03.525444 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:03.525401 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" event={"ID":"70a3efb9-7e77-4ed8-a69b-25f7d1446b19","Type":"ContainerDied","Data":"194652d88e5f84453d84c7a25ad1b72da280d203dce81d5ca1ad07f6acd5dd8b"} Oct 30 18:38:03.525774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:03.525742 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="194652d88e5f84453d84c7a25ad1b72da280d203dce81d5ca1ad07f6acd5dd8b" Oct 30 18:38:03.525976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:03.525959 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj" Oct 30 18:38:03.532655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:03.532620 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" event={"ID":"64f9a074-8927-4f97-9287-6105c91e6bfb","Type":"ContainerDied","Data":"3767c0da682db259d98af49db2af371b74158772a844d8c6e66d7975b1ffacf2"} Oct 30 18:38:03.534460 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:03.533140 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3767c0da682db259d98af49db2af371b74158772a844d8c6e66d7975b1ffacf2" Oct 30 18:38:03.534717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:03.534698 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r" Oct 30 18:38:04.418961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:04.416095 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:04.418961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:04.416779 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:04.470958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:04.468050 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:38:05.372266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:05.281291 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:05.372266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:05.287999 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:38:05.372266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:05.341215 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:38:05.372266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:05.368005 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:05.591408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:05.579078 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 18:38:05.640339 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:05.630917 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:05.828310 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:38:05.828272 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:38:05.851101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:05.851054 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:06.299959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.288392 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Oct 30 18:38:06.299959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.289009 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:38:06.351964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.339185 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:06.351964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.339758 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:38:06.352364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.352321 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:06.353101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.353077 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:38:06.373100 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.373060 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:38:06.373823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.373800 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:38:06.454955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.420024 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:06.454955 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.420603 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": context deadline exceeded" Oct 30 18:38:06.455334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.455300 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:06.605411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.593720 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:06.690088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.690040 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:06.691115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.691091 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:38:06.709025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.658455 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:38:06.728094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.709715 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:38:06.955026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.952828 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:06.979156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:06.975833 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:38:07.030257 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.007020 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:07.035754 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.035702 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" podUID="0086e9f0-1201-4263-b710-5417225349ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.479405 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.479356 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.503417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.503373 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:38:07.525091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.525041 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.564318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.542193 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.576966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.551271 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.601960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.553354 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.603373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.553379 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:38:07.706092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.692759 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:07.754428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.745103 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 18:38:07.946957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.941641 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:38:07.946957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.941705 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:38:07.946957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:07.941745 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:38:08.082011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.005764 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 30 18:38:08.296783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.143417 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:08.296783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.291599 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.296783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.270163 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.296783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.195995 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd"] Oct 30 18:38:08.296783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.292540 4883 cpu_manager.go:414] "containerMap: removing container" podUID="70a3efb9-7e77-4ed8-a69b-25f7d1446b19" containerName="revoke-keys" Oct 30 18:38:08.296783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.292577 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a3efb9-7e77-4ed8-a69b-25f7d1446b19" containerName="revoke-keys" Oct 30 18:38:08.296783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.292627 4883 cpu_manager.go:414] "containerMap: removing container" podUID="64f9a074-8927-4f97-9287-6105c91e6bfb" containerName="revoke-keys" Oct 30 18:38:08.296783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.292638 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f9a074-8927-4f97-9287-6105c91e6bfb" containerName="revoke-keys" Oct 30 18:38:08.296783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.292786 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a3efb9-7e77-4ed8-a69b-25f7d1446b19" containerName="revoke-keys" Oct 30 18:38:08.296783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.292809 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f9a074-8927-4f97-9287-6105c91e6bfb" containerName="revoke-keys" Oct 30 18:38:08.297602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.201624 4883 patch_prober.go:29] interesting pod/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:08.324078 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.324033 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.327489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.324413 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx"] Oct 30 18:38:08.347552 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.324550 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:08.408303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.408268 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:08.425979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.419385 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:38:08.425979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.419479 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:38:08.425979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.419547 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:38:08.425979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.419624 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:38:08.426597 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.426570 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c3ebfb-tt6wd\" (UID: \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:08.429958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.426833 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9cd\" (UniqueName: \"kubernetes.io/projected/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-kube-api-access-nj9cd\") pod \"maas-api-revoke-sub-71c3ebfb-tt6wd\" (UID: \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:08.435960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.430292 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c3ebfb-tt6wd\" (UID: \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:08.460173 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.460127 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.522959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.503840 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.523198 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.512503 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": context deadline exceeded" Oct 30 18:38:08.576497 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.575325 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:38:08.580721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.576748 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 18:38:08.596956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.582018 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.596956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.582175 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9cd\" (UniqueName: \"kubernetes.io/projected/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-kube-api-access-nj9cd\") pod \"maas-api-revoke-sub-71c3ebfb-tt6wd\" (UID: \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:08.596956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.582225 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-744af70c-2xgtx\" (UID: \"c263b264-7bcd-4bcb-a8b1-f81d46374867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:08.596956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.582349 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c3ebfb-tt6wd\" (UID: \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:08.596956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.582419 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk7j4\" (UniqueName: \"kubernetes.io/projected/c263b264-7bcd-4bcb-a8b1-f81d46374867-kube-api-access-bk7j4\") pod \"maas-api-revoke-sub-744af70c-2xgtx\" (UID: \"c263b264-7bcd-4bcb-a8b1-f81d46374867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:08.596956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.582461 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-744af70c-2xgtx\" (UID: \"c263b264-7bcd-4bcb-a8b1-f81d46374867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:08.596956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.582614 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c3ebfb-tt6wd\" (UID: \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:08.607113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.607006 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:38:08.636857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.622062 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:38:08.636857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.624020 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c3ebfb-tt6wd\" (UID: \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:08.674961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.669815 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd"] Oct 30 18:38:08.735957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.729493 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-744af70c-2xgtx\" (UID: \"c263b264-7bcd-4bcb-a8b1-f81d46374867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:08.735957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.729691 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk7j4\" (UniqueName: \"kubernetes.io/projected/c263b264-7bcd-4bcb-a8b1-f81d46374867-kube-api-access-bk7j4\") pod \"maas-api-revoke-sub-744af70c-2xgtx\" (UID: \"c263b264-7bcd-4bcb-a8b1-f81d46374867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:08.735957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.729738 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-744af70c-2xgtx\" (UID: \"c263b264-7bcd-4bcb-a8b1-f81d46374867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:08.760958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.750103 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-744af70c-2xgtx\" (UID: \"c263b264-7bcd-4bcb-a8b1-f81d46374867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:08.764688 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.764643 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.784879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.772175 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx"] Oct 30 18:38:08.785137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.784880 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9cd\" (UniqueName: \"kubernetes.io/projected/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-kube-api-access-nj9cd\") pod \"maas-api-revoke-sub-71c3ebfb-tt6wd\" (UID: \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:08.785737 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.785701 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:08.812425 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:08.812389 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c3ebfb-tt6wd\" (UID: \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:09.059960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.056037 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.088957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.087068 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-m2fn9" podUID="50a26671-57c2-4781-9b78-211ea23c0318" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.150050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.117586 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:09.167453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.167419 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:38:09.209959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.186511 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk7j4\" (UniqueName: \"kubernetes.io/projected/c263b264-7bcd-4bcb-a8b1-f81d46374867-kube-api-access-bk7j4\") pod \"maas-api-revoke-sub-744af70c-2xgtx\" (UID: \"c263b264-7bcd-4bcb-a8b1-f81d46374867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:09.231013 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.219769 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-744af70c-2xgtx\" (UID: \"c263b264-7bcd-4bcb-a8b1-f81d46374867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:09.284008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.279584 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:09.284550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.284516 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.448690 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.447482 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:38:09.525059 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.525012 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.535965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.525281 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" containerID="cri-o://0a6d2485d783be3c58f1ea8390c413342934380c4bdf352ca018fbf800fdc2c8" gracePeriod=45 Oct 30 18:38:09.582022 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.581685 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.582188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.582101 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:38:09.582408 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.582376 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.599716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.591899 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" containerID="cri-o://ad2c56d6ae3710a4bc8370d34269391b549cc60123cf09e81eafb0bfc6e47b21" gracePeriod=45 Oct 30 18:38:09.600770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.600733 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" containerID="cri-o://44b9673f6528e93cd3b0176a1e8c49e207f2dcb3ff7cc36119476177c8d59b15" gracePeriod=45 Oct 30 18:38:09.617009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.614243 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:38:09.626050 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.626008 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.757366 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.757022 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:09.852511 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:09.852463 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.279963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.268802 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": read tcp 10.132.0.2:49168->10.132.0.49:8000: read: connection reset by peer" Oct 30 18:38:10.279963 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.274276 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": dial tcp 10.132.0.49:8000: connect: connection refused" Oct 30 18:38:10.287807 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.286652 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.576918 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.576810 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 18:38:10.599224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.582020 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.599224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.582226 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.599224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.577221 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.599224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.577247 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.599661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.577188 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": dial tcp 10.132.0.53:8000: connect: connection refused" Oct 30 18:38:10.625995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.622292 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": dial tcp 10.132.0.84:8000: connect: connection refused" Oct 30 18:38:10.625995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.622785 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.629465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.629430 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.873039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.872028 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:10.974005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:10.973592 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx"] Oct 30 18:38:11.400418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.251336 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:11.400418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.251419 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.400418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.271974 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:11.400418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.272056 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.532967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.532339 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:38:11.587796 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.564176 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": dial tcp 10.132.0.84:8000: connect: connection refused" Oct 30 18:38:11.587796 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.564623 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": dial tcp 10.132.0.49:8000: connect: connection refused" Oct 30 18:38:11.615129 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.590161 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.615129 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.610972 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.689027 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.670037 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.689027 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.684338 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:11.706134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.697295 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 30 18:38:11.803294 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.739389 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": dial tcp 10.132.0.53:8000: connect: connection refused" Oct 30 18:38:11.872972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.849209 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:38:11.872972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.849267 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd"] Oct 30 18:38:11.872972 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.849372 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:38:11.935018 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:11.908122 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:12.262801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.239840 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": dial tcp 10.132.0.49:8000: connect: connection refused" Oct 30 18:38:12.413956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.403718 4883 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:12.479738 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.479692 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": dial tcp 10.132.0.84:8000: connect: connection refused" Oct 30 18:38:12.531791 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.530553 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:38:12.563977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.540216 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:12.740243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.739965 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 18:38:12.743770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.743466 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:12.752106 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.752072 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 18:38:12.753098 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.753055 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:38:12.802200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.802161 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": dial tcp 10.132.0.53:8000: connect: connection refused" Oct 30 18:38:12.827867 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.827832 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:38:12.829002 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.828976 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:38:12.830087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.830060 4883 generic.go:356] "Generic (PLEG): container finished" podID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerID="ad2c56d6ae3710a4bc8370d34269391b549cc60123cf09e81eafb0bfc6e47b21" exitCode=0 Oct 30 18:38:12.830292 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.830274 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" event={"ID":"d8eca37e-ab66-4007-b362-00c0359a2c07","Type":"ContainerDied","Data":"ad2c56d6ae3710a4bc8370d34269391b549cc60123cf09e81eafb0bfc6e47b21"} Oct 30 18:38:12.831330 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.831303 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:38:12.831429 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.831353 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:38:12.833611 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.833577 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" event={"ID":"c263b264-7bcd-4bcb-a8b1-f81d46374867","Type":"ContainerStarted","Data":"cb4a2c9d661203c4388f6bf0aae330582bbcbf7dc14d2c1c8dec357e7fb7db93"} Oct 30 18:38:12.836056 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.836006 4883 generic.go:356] "Generic (PLEG): container finished" podID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerID="44b9673f6528e93cd3b0176a1e8c49e207f2dcb3ff7cc36119476177c8d59b15" exitCode=0 Oct 30 18:38:12.836133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.836108 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" event={"ID":"f16a36bd-5d13-496e-bbef-965ca65aec24","Type":"ContainerDied","Data":"44b9673f6528e93cd3b0176a1e8c49e207f2dcb3ff7cc36119476177c8d59b15"} Oct 30 18:38:12.838160 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.838127 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" event={"ID":"b16bf630-ad79-492f-ad96-cf2cdfad0bb3","Type":"ContainerStarted","Data":"8f455b59ae5599a00250eee2b8c3a14d669de579e4d96cfe15a64f830687a1f0"} Oct 30 18:38:12.839859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.839838 4883 generic.go:356] "Generic (PLEG): container finished" podID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerID="0a6d2485d783be3c58f1ea8390c413342934380c4bdf352ca018fbf800fdc2c8" exitCode=0 Oct 30 18:38:12.839929 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:12.839903 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" event={"ID":"508b44e7-8aef-4e22-84af-8efaa2b9c5d6","Type":"ContainerDied","Data":"0a6d2485d783be3c58f1ea8390c413342934380c4bdf352ca018fbf800fdc2c8"} Oct 30 18:38:13.125889 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.125854 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:38:13.131021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.130994 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:38:13.135410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.135377 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:38:13.236947 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.236897 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-dshm\") pod \"d8eca37e-ab66-4007-b362-00c0359a2c07\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " Oct 30 18:38:13.237139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.236994 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-model-cache\") pod \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " Oct 30 18:38:13.237139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237049 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-home\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-home\") pod \"d8eca37e-ab66-4007-b362-00c0359a2c07\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " Oct 30 18:38:13.237139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237079 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16a36bd-5d13-496e-bbef-965ca65aec24-kube-api-access-qfvmp\" (UniqueName: \"kubernetes.io/projected/f16a36bd-5d13-496e-bbef-965ca65aec24-kube-api-access-qfvmp\") pod \"f16a36bd-5d13-496e-bbef-965ca65aec24\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " Oct 30 18:38:13.237764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237137 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-home\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-home\") pod \"f16a36bd-5d13-496e-bbef-965ca65aec24\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " Oct 30 18:38:13.237764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237194 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-home\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-home\") pod \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " Oct 30 18:38:13.237764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237265 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-tmp-dir\") pod \"f16a36bd-5d13-496e-bbef-965ca65aec24\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " Oct 30 18:38:13.237764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237300 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-tmp-dir\") pod \"d8eca37e-ab66-4007-b362-00c0359a2c07\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " Oct 30 18:38:13.237764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237342 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16a36bd-5d13-496e-bbef-965ca65aec24-tls-certs\" (UniqueName: \"kubernetes.io/secret/f16a36bd-5d13-496e-bbef-965ca65aec24-tls-certs\") pod \"f16a36bd-5d13-496e-bbef-965ca65aec24\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " Oct 30 18:38:13.237764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237370 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-kube-api-access-jh2h2\" (UniqueName: \"kubernetes.io/projected/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-kube-api-access-jh2h2\") pod \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " Oct 30 18:38:13.237764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237484 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-dshm\") pod \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " Oct 30 18:38:13.237764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237518 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tls-certs\") pod \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " Oct 30 18:38:13.237764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237565 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tmp-dir\") pod \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\" (UID: \"508b44e7-8aef-4e22-84af-8efaa2b9c5d6\") " Oct 30 18:38:13.237764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237614 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-model-cache\") pod \"d8eca37e-ab66-4007-b362-00c0359a2c07\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " Oct 30 18:38:13.237764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237683 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8eca37e-ab66-4007-b362-00c0359a2c07-kube-api-access-zkxfj\" (UniqueName: \"kubernetes.io/projected/d8eca37e-ab66-4007-b362-00c0359a2c07-kube-api-access-zkxfj\") pod \"d8eca37e-ab66-4007-b362-00c0359a2c07\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " Oct 30 18:38:13.237764 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237721 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-model-cache\") pod \"f16a36bd-5d13-496e-bbef-965ca65aec24\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " Oct 30 18:38:13.238432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237789 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8eca37e-ab66-4007-b362-00c0359a2c07-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8eca37e-ab66-4007-b362-00c0359a2c07-tls-certs\") pod \"d8eca37e-ab66-4007-b362-00c0359a2c07\" (UID: \"d8eca37e-ab66-4007-b362-00c0359a2c07\") " Oct 30 18:38:13.238432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.237829 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-dshm\") pod \"f16a36bd-5d13-496e-bbef-965ca65aec24\" (UID: \"f16a36bd-5d13-496e-bbef-965ca65aec24\") " Oct 30 18:38:13.239090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.239064 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tmp-dir" pod "508b44e7-8aef-4e22-84af-8efaa2b9c5d6" (UID: "508b44e7-8aef-4e22-84af-8efaa2b9c5d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.239713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.239689 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-model-cache" pod "d8eca37e-ab66-4007-b362-00c0359a2c07" (UID: "d8eca37e-ab66-4007-b362-00c0359a2c07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.240526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.240500 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-model-cache" pod "508b44e7-8aef-4e22-84af-8efaa2b9c5d6" (UID: "508b44e7-8aef-4e22-84af-8efaa2b9c5d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.240526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.240504 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-model-cache" pod "f16a36bd-5d13-496e-bbef-965ca65aec24" (UID: "f16a36bd-5d13-496e-bbef-965ca65aec24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.240708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.240672 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-home" pod "508b44e7-8aef-4e22-84af-8efaa2b9c5d6" (UID: "508b44e7-8aef-4e22-84af-8efaa2b9c5d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.240708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.240678 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-tmp-dir" pod "d8eca37e-ab66-4007-b362-00c0359a2c07" (UID: "d8eca37e-ab66-4007-b362-00c0359a2c07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.241146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.241073 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-dshm" pod "f16a36bd-5d13-496e-bbef-965ca65aec24" (UID: "f16a36bd-5d13-496e-bbef-965ca65aec24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.241233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.241145 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-home" pod "d8eca37e-ab66-4007-b362-00c0359a2c07" (UID: "d8eca37e-ab66-4007-b362-00c0359a2c07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.241233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.241190 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16a36bd-5d13-496e-bbef-965ca65aec24-tls-certs" pod "f16a36bd-5d13-496e-bbef-965ca65aec24" (UID: "f16a36bd-5d13-496e-bbef-965ca65aec24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:13.241467 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.241444 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-tmp-dir" pod "f16a36bd-5d13-496e-bbef-965ca65aec24" (UID: "f16a36bd-5d13-496e-bbef-965ca65aec24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.241581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.241563 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-home" pod "f16a36bd-5d13-496e-bbef-965ca65aec24" (UID: "f16a36bd-5d13-496e-bbef-965ca65aec24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.241713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.241696 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-kube-api-access-jh2h2" pod "508b44e7-8aef-4e22-84af-8efaa2b9c5d6" (UID: "508b44e7-8aef-4e22-84af-8efaa2b9c5d6"). InnerVolumeSpecName "kube-api-access-jh2h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:13.242089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.242062 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-dshm" pod "d8eca37e-ab66-4007-b362-00c0359a2c07" (UID: "d8eca37e-ab66-4007-b362-00c0359a2c07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.242823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.242801 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8eca37e-ab66-4007-b362-00c0359a2c07-kube-api-access-zkxfj" pod "d8eca37e-ab66-4007-b362-00c0359a2c07" (UID: "d8eca37e-ab66-4007-b362-00c0359a2c07"). InnerVolumeSpecName "kube-api-access-zkxfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:13.244082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.244053 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8eca37e-ab66-4007-b362-00c0359a2c07-tls-certs" pod "d8eca37e-ab66-4007-b362-00c0359a2c07" (UID: "d8eca37e-ab66-4007-b362-00c0359a2c07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:13.244082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.244068 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-dshm" pod "508b44e7-8aef-4e22-84af-8efaa2b9c5d6" (UID: "508b44e7-8aef-4e22-84af-8efaa2b9c5d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:13.244204 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.244071 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16a36bd-5d13-496e-bbef-965ca65aec24-kube-api-access-qfvmp" pod "f16a36bd-5d13-496e-bbef-965ca65aec24" (UID: "f16a36bd-5d13-496e-bbef-965ca65aec24"). InnerVolumeSpecName "kube-api-access-qfvmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:13.244469 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.244444 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tls-certs" pod "508b44e7-8aef-4e22-84af-8efaa2b9c5d6" (UID: "508b44e7-8aef-4e22-84af-8efaa2b9c5d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:13.340322 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340235 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340322 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340292 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340322 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340310 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340322 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340324 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340338 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkxfj\" (UniqueName: \"kubernetes.io/projected/d8eca37e-ab66-4007-b362-00c0359a2c07-kube-api-access-zkxfj\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340370 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340387 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8eca37e-ab66-4007-b362-00c0359a2c07-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340411 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340423 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340446 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340459 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340472 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfvmp\" (UniqueName: \"kubernetes.io/projected/f16a36bd-5d13-496e-bbef-965ca65aec24-kube-api-access-qfvmp\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340487 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340520 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340543 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f16a36bd-5d13-496e-bbef-965ca65aec24-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340559 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8eca37e-ab66-4007-b362-00c0359a2c07-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340573 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16a36bd-5d13-496e-bbef-965ca65aec24-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.340630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.340587 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2h2\" (UniqueName: \"kubernetes.io/projected/508b44e7-8aef-4e22-84af-8efaa2b9c5d6-kube-api-access-jh2h2\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.851012 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.850972 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" event={"ID":"f16a36bd-5d13-496e-bbef-965ca65aec24","Type":"ContainerDied","Data":"e827cde89f0e3806920640cae1caecd366111e3a08ccd2afe5f500b28361b3c9"} Oct 30 18:38:13.851012 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.851002 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl" Oct 30 18:38:13.851261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.851039 4883 scope.go:122] "RemoveContainer" containerID="44b9673f6528e93cd3b0176a1e8c49e207f2dcb3ff7cc36119476177c8d59b15" Oct 30 18:38:13.853083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.853062 4883 generic.go:356] "Generic (PLEG): container finished" podID="b16bf630-ad79-492f-ad96-cf2cdfad0bb3" containerID="28c134b0304a6694047d20226b990b41ea5ee771602d11d17181d705a2856e4d" exitCode=0 Oct 30 18:38:13.853190 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.853140 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" event={"ID":"b16bf630-ad79-492f-ad96-cf2cdfad0bb3","Type":"ContainerDied","Data":"28c134b0304a6694047d20226b990b41ea5ee771602d11d17181d705a2856e4d"} Oct 30 18:38:13.854847 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.854827 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" event={"ID":"508b44e7-8aef-4e22-84af-8efaa2b9c5d6","Type":"ContainerDied","Data":"c6689a707a0ef05c975a8847f54d507d6fe6f833ad86a050bdedd7ab5ec3d343"} Oct 30 18:38:13.854988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.854897 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb" Oct 30 18:38:13.858075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.858051 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" event={"ID":"d8eca37e-ab66-4007-b362-00c0359a2c07","Type":"ContainerDied","Data":"1d0bba59153943406eebc202eef1d1ea2dfe30b66808b20c2d8e50d773286194"} Oct 30 18:38:13.858196 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.858133 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp" Oct 30 18:38:13.884248 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.882457 4883 scope.go:122] "RemoveContainer" containerID="0a6d2485d783be3c58f1ea8390c413342934380c4bdf352ca018fbf800fdc2c8" Oct 30 18:38:13.892324 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.890689 4883 generic.go:356] "Generic (PLEG): container finished" podID="c263b264-7bcd-4bcb-a8b1-f81d46374867" containerID="70d823ba563dec50eb424fe37bdd279ee893bd9f2e6a7c504979b272a46c9555" exitCode=0 Oct 30 18:38:13.892324 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.890744 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" event={"ID":"c263b264-7bcd-4bcb-a8b1-f81d46374867","Type":"ContainerDied","Data":"70d823ba563dec50eb424fe37bdd279ee893bd9f2e6a7c504979b272a46c9555"} Oct 30 18:38:13.911900 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:13.911871 4883 scope.go:122] "RemoveContainer" containerID="ad2c56d6ae3710a4bc8370d34269391b549cc60123cf09e81eafb0bfc6e47b21" Oct 30 18:38:14.044693 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:14.044657 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl"] Oct 30 18:38:14.048559 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:14.048526 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/simulator-w2-6e8fca-kserve-54ff9d895b-qf8dl"] Oct 30 18:38:14.094723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:14.094685 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb"] Oct 30 18:38:14.100302 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:14.100270 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/distinct-w2-6e8fca-kserve-5969cc768d-tn7hb"] Oct 30 18:38:14.198200 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:14.198096 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp"] Oct 30 18:38:14.202244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:14.202209 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6e8fca/premium-w2-6e8fca-kserve-7cd794c749-dzwbp"] Oct 30 18:38:15.211999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.205073 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" path="/var/lib/kubelet/pods/508b44e7-8aef-4e22-84af-8efaa2b9c5d6/volumes" Oct 30 18:38:15.211999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.205798 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" path="/var/lib/kubelet/pods/d8eca37e-ab66-4007-b362-00c0359a2c07/volumes" Oct 30 18:38:15.211999 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.206432 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" path="/var/lib/kubelet/pods/f16a36bd-5d13-496e-bbef-965ca65aec24/volumes" Oct 30 18:38:15.433970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.418162 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:15.433970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.428824 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-xks-ca\") pod \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\" (UID: \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\") " Oct 30 18:38:15.433970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.428983 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-openshift-ca\") pod \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\" (UID: \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\") " Oct 30 18:38:15.433970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.429124 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-kube-api-access-nj9cd\" (UniqueName: \"kubernetes.io/projected/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-kube-api-access-nj9cd\") pod \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\" (UID: \"b16bf630-ad79-492f-ad96-cf2cdfad0bb3\") " Oct 30 18:38:15.433970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.429502 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:15.452160 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.451079 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-openshift-ca" pod "b16bf630-ad79-492f-ad96-cf2cdfad0bb3" (UID: "b16bf630-ad79-492f-ad96-cf2cdfad0bb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:15.452160 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.451763 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:15.490008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.487285 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-xks-ca" pod "b16bf630-ad79-492f-ad96-cf2cdfad0bb3" (UID: "b16bf630-ad79-492f-ad96-cf2cdfad0bb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:15.493958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.493519 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-kube-api-access-nj9cd" pod "b16bf630-ad79-492f-ad96-cf2cdfad0bb3" (UID: "b16bf630-ad79-492f-ad96-cf2cdfad0bb3"). InnerVolumeSpecName "kube-api-access-nj9cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:15.610965 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.596134 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-xks-ca\") pod \"c263b264-7bcd-4bcb-a8b1-f81d46374867\" (UID: \"c263b264-7bcd-4bcb-a8b1-f81d46374867\") " Oct 30 18:38:15.611423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.611395 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c263b264-7bcd-4bcb-a8b1-f81d46374867-kube-api-access-bk7j4\" (UniqueName: \"kubernetes.io/projected/c263b264-7bcd-4bcb-a8b1-f81d46374867-kube-api-access-bk7j4\") pod \"c263b264-7bcd-4bcb-a8b1-f81d46374867\" (UID: \"c263b264-7bcd-4bcb-a8b1-f81d46374867\") " Oct 30 18:38:15.619894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.618150 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-openshift-ca\") pod \"c263b264-7bcd-4bcb-a8b1-f81d46374867\" (UID: \"c263b264-7bcd-4bcb-a8b1-f81d46374867\") " Oct 30 18:38:15.619894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.619012 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj9cd\" (UniqueName: \"kubernetes.io/projected/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-kube-api-access-nj9cd\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:15.619894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.619040 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16bf630-ad79-492f-ad96-cf2cdfad0bb3-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:15.625987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.622650 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-openshift-ca" pod "c263b264-7bcd-4bcb-a8b1-f81d46374867" (UID: "c263b264-7bcd-4bcb-a8b1-f81d46374867"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:15.646043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.644505 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c263b264-7bcd-4bcb-a8b1-f81d46374867-kube-api-access-bk7j4" pod "c263b264-7bcd-4bcb-a8b1-f81d46374867" (UID: "c263b264-7bcd-4bcb-a8b1-f81d46374867"). InnerVolumeSpecName "kube-api-access-bk7j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:15.646043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.644595 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-xks-ca" pod "c263b264-7bcd-4bcb-a8b1-f81d46374867" (UID: "c263b264-7bcd-4bcb-a8b1-f81d46374867"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:15.720311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.720275 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:15.720311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.720310 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk7j4\" (UniqueName: \"kubernetes.io/projected/c263b264-7bcd-4bcb-a8b1-f81d46374867-kube-api-access-bk7j4\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:15.720568 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.720326 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c263b264-7bcd-4bcb-a8b1-f81d46374867-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:15.917418 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.917390 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" Oct 30 18:38:15.917599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.917415 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx" event={"ID":"c263b264-7bcd-4bcb-a8b1-f81d46374867","Type":"ContainerDied","Data":"cb4a2c9d661203c4388f6bf0aae330582bbcbf7dc14d2c1c8dec357e7fb7db93"} Oct 30 18:38:15.917599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.917487 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb4a2c9d661203c4388f6bf0aae330582bbcbf7dc14d2c1c8dec357e7fb7db93" Oct 30 18:38:15.926497 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.926470 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" Oct 30 18:38:15.926626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.926516 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd" event={"ID":"b16bf630-ad79-492f-ad96-cf2cdfad0bb3","Type":"ContainerDied","Data":"8f455b59ae5599a00250eee2b8c3a14d669de579e4d96cfe15a64f830687a1f0"} Oct 30 18:38:15.926626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:15.926579 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f455b59ae5599a00250eee2b8c3a14d669de579e4d96cfe15a64f830687a1f0" Oct 30 18:38:18.158285 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158243 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8"] Oct 30 18:38:18.158823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158804 4883 cpu_manager.go:414] "containerMap: removing container" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" Oct 30 18:38:18.158823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158815 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" Oct 30 18:38:18.158983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158835 4883 cpu_manager.go:414] "containerMap: removing container" podUID="b16bf630-ad79-492f-ad96-cf2cdfad0bb3" containerName="revoke-keys" Oct 30 18:38:18.158983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158840 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16bf630-ad79-492f-ad96-cf2cdfad0bb3" containerName="revoke-keys" Oct 30 18:38:18.158983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158850 4883 cpu_manager.go:414] "containerMap: removing container" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" Oct 30 18:38:18.158983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158857 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" Oct 30 18:38:18.158983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158876 4883 cpu_manager.go:414] "containerMap: removing container" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" Oct 30 18:38:18.158983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158881 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" Oct 30 18:38:18.158983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158889 4883 cpu_manager.go:414] "containerMap: removing container" podUID="c263b264-7bcd-4bcb-a8b1-f81d46374867" containerName="revoke-keys" Oct 30 18:38:18.158983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158894 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c263b264-7bcd-4bcb-a8b1-f81d46374867" containerName="revoke-keys" Oct 30 18:38:18.158983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158985 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="508b44e7-8aef-4e22-84af-8efaa2b9c5d6" containerName="main" Oct 30 18:38:18.159477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.158997 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16bf630-ad79-492f-ad96-cf2cdfad0bb3" containerName="revoke-keys" Oct 30 18:38:18.159477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.159005 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16a36bd-5d13-496e-bbef-965ca65aec24" containerName="main" Oct 30 18:38:18.159477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.159014 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8eca37e-ab66-4007-b362-00c0359a2c07" containerName="main" Oct 30 18:38:18.159477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.159020 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="c263b264-7bcd-4bcb-a8b1-f81d46374867" containerName="revoke-keys" Oct 30 18:38:18.163197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.163171 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.182537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.182489 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a1a31f\"/\"explicit-model-a1a31f-kserve-self-signed-certs\"" Oct 30 18:38:18.196307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.196227 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a1a31f\"/\"default-dockercfg-qhvn7\"" Oct 30 18:38:18.196307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.196255 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a1a31f\"/\"kube-root-ca.crt\"" Oct 30 18:38:18.196606 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.196406 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a1a31f\"/\"openshift-service-ca.crt\"" Oct 30 18:38:18.205981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.201276 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8"] Oct 30 18:38:18.260416 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.260385 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-model-cache\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.260416 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.260429 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b87546-1283-431a-b62a-2718b90bc3cb-tls-certs\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.260673 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.260642 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-dshm\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.260739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.260688 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-tmp-dir\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.260811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.260780 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wvll\" (UniqueName: \"kubernetes.io/projected/85b87546-1283-431a-b62a-2718b90bc3cb-kube-api-access-4wvll\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.265077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.260820 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-home\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.362616 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.362578 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-model-cache\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.362800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.362640 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b87546-1283-431a-b62a-2718b90bc3cb-tls-certs\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.362800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.362789 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-dshm\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.362979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.362818 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-tmp-dir\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.362979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.362901 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wvll\" (UniqueName: \"kubernetes.io/projected/85b87546-1283-431a-b62a-2718b90bc3cb-kube-api-access-4wvll\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.363098 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.362980 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-home\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.363098 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.363020 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-model-cache\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.363509 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.363298 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-tmp-dir\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.363509 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.363463 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-home\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.365303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.365276 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-dshm\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.365598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.365573 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b87546-1283-431a-b62a-2718b90bc3cb-tls-certs\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.432036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.426644 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wvll\" (UniqueName: \"kubernetes.io/projected/85b87546-1283-431a-b62a-2718b90bc3cb-kube-api-access-4wvll\") pod \"explicit-model-a1a31f-kserve-59474bc57-lwvf8\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.479231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.479188 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:18.932712 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.932688 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8"] Oct 30 18:38:18.935117 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:38:18.935086 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85b87546_1283_431a_b62a_2718b90bc3cb.slice/crio-3826c2f031cbf569181e595b10c78eed9b1dc942baf43f56f9a037ae518ebced WatchSource:0}: Error finding container 3826c2f031cbf569181e595b10c78eed9b1dc942baf43f56f9a037ae518ebced: Status 404 returned error can't find the container with id 3826c2f031cbf569181e595b10c78eed9b1dc942baf43f56f9a037ae518ebced Oct 30 18:38:18.957007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:18.956979 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" event={"ID":"85b87546-1283-431a-b62a-2718b90bc3cb","Type":"ContainerStarted","Data":"3826c2f031cbf569181e595b10c78eed9b1dc942baf43f56f9a037ae518ebced"} Oct 30 18:38:20.013035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:20.011564 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" event={"ID":"85b87546-1283-431a-b62a-2718b90bc3cb","Type":"ContainerStarted","Data":"5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d"} Oct 30 18:38:20.015766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:20.015738 4883 generic.go:356] "Generic (PLEG): container finished" podID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerID="20da60c9571f2a60babdc1170cc039a0f239d2b0f5e116c2bb210a06900a642d" exitCode=0 Oct 30 18:38:20.015974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:20.015777 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" event={"ID":"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470","Type":"ContainerDied","Data":"20da60c9571f2a60babdc1170cc039a0f239d2b0f5e116c2bb210a06900a642d"} Oct 30 18:38:20.016292 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:20.016271 4883 scope.go:122] "RemoveContainer" containerID="20da60c9571f2a60babdc1170cc039a0f239d2b0f5e116c2bb210a06900a642d" Oct 30 18:38:20.018134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:20.018101 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:38:20.018253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:20.018183 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:38:20.018253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:20.018240 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:38:20.127306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:20.127241 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" podStartSLOduration=2.127221368 podStartE2EDuration="2.127221368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:20.122483532 +0000 UTC m=+2759.079487367" watchObservedRunningTime="2026-10-30 18:38:20.127221368 +0000 UTC m=+2759.084225205" Oct 30 18:38:20.480560 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:20.480525 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:21.032598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:21.032550 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" event={"ID":"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470","Type":"ContainerStarted","Data":"20a2f17b46487cc0b08c1199b789b708a6f3cb4e6e40323494f9f6e145984fd7"} Oct 30 18:38:21.210761 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:21.210730 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:38:21.210953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:21.210919 4883 patch_prober.go:29] interesting pod/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 18:38:21.211011 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:21.210978 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 18:38:22.211523 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:22.211481 4883 patch_prober.go:29] interesting pod/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 18:38:22.211909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:22.211543 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 18:38:23.211362 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:23.211325 4883 patch_prober.go:29] interesting pod/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 18:38:23.211561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:23.211389 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 18:38:24.212013 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:24.211975 4883 patch_prober.go:29] interesting pod/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 18:38:24.212459 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:24.212037 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 18:38:25.210869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:25.210830 4883 patch_prober.go:29] interesting pod/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 18:38:25.211084 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:25.210894 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 18:38:26.211238 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:26.211199 4883 patch_prober.go:29] interesting pod/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 18:38:26.211656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:26.211270 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 18:38:27.216625 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:27.216593 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:38:27.217120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:27.217046 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:38:27.217890 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:27.217868 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:38:28.480365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:28.480323 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:28.493507 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:28.493482 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:29.186379 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:29.186348 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:32.128039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.128002 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j"] Oct 30 18:38:32.493800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.493717 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5"] Oct 30 18:38:32.499426 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.499402 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:32.518752 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.518727 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:38:32.519086 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.519068 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:38:32.542828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.542793 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:38:32.543070 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.542850 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:38:32.545295 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.545272 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5"] Oct 30 18:38:32.651913 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.651873 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71abc140-6296-4993-b427-61a9977cabde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-58d08b-m8nn5\" (UID: \"71abc140-6296-4993-b427-61a9977cabde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:32.656185 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.652511 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8"] Oct 30 18:38:32.656781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.656306 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71abc140-6296-4993-b427-61a9977cabde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-58d08b-m8nn5\" (UID: \"71abc140-6296-4993-b427-61a9977cabde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:32.657140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.657120 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkmh6\" (UniqueName: \"kubernetes.io/projected/71abc140-6296-4993-b427-61a9977cabde-kube-api-access-gkmh6\") pod \"maas-api-revoke-keys-e2e-autores-58d08b-m8nn5\" (UID: \"71abc140-6296-4993-b427-61a9977cabde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:32.782224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.781601 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71abc140-6296-4993-b427-61a9977cabde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-58d08b-m8nn5\" (UID: \"71abc140-6296-4993-b427-61a9977cabde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:32.782627 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.782600 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkmh6\" (UniqueName: \"kubernetes.io/projected/71abc140-6296-4993-b427-61a9977cabde-kube-api-access-gkmh6\") pod \"maas-api-revoke-keys-e2e-autores-58d08b-m8nn5\" (UID: \"71abc140-6296-4993-b427-61a9977cabde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:32.782924 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.782905 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71abc140-6296-4993-b427-61a9977cabde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-58d08b-m8nn5\" (UID: \"71abc140-6296-4993-b427-61a9977cabde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:32.801196 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.801160 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71abc140-6296-4993-b427-61a9977cabde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-58d08b-m8nn5\" (UID: \"71abc140-6296-4993-b427-61a9977cabde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:32.836008 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.827497 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkmh6\" (UniqueName: \"kubernetes.io/projected/71abc140-6296-4993-b427-61a9977cabde-kube-api-access-gkmh6\") pod \"maas-api-revoke-keys-e2e-autores-58d08b-m8nn5\" (UID: \"71abc140-6296-4993-b427-61a9977cabde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:32.851961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:32.843615 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71abc140-6296-4993-b427-61a9977cabde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-58d08b-m8nn5\" (UID: \"71abc140-6296-4993-b427-61a9977cabde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:33.064671 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.064580 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7"] Oct 30 18:38:33.068979 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.068953 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:33.086821 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.086790 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7\" (UID: \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:33.087021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.086829 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx9gs\" (UniqueName: \"kubernetes.io/projected/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-kube-api-access-jx9gs\") pod \"maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7\" (UID: \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:33.087021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.086980 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7\" (UID: \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:33.105051 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.105008 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7"] Oct 30 18:38:33.113989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.113962 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:33.188446 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.188408 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7\" (UID: \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:33.188852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.188472 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx9gs\" (UniqueName: \"kubernetes.io/projected/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-kube-api-access-jx9gs\") pod \"maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7\" (UID: \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:33.188852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.188547 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7\" (UID: \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:33.189245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.189195 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7\" (UID: \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:33.196537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.196503 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7\" (UID: \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:33.218157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.218122 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx9gs\" (UniqueName: \"kubernetes.io/projected/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-kube-api-access-jx9gs\") pod \"maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7\" (UID: \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:33.230091 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.229991 4883 scope.go:122] "RemoveContainer" containerID="8e63d2e7f63675b60692c1063bd49de20437dbe1fbf700368fa63ed7f8759725" Oct 30 18:38:33.243526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.243499 4883 scope.go:122] "RemoveContainer" containerID="5b9f3f24682b30bec85bbeec83f125b837a4d39442c056978197138b91420e32" Oct 30 18:38:33.383895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.383863 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:33.546888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.546795 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5"] Oct 30 18:38:33.731490 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:38:33.731460 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52b900a_f62c_45e5_bc7b_b3bb1afb7075.slice/crio-8f1471d5aa8cdb387116c77d6dc9ce1d0d5938138d4915a67ba3ec267f9691f2 WatchSource:0}: Error finding container 8f1471d5aa8cdb387116c77d6dc9ce1d0d5938138d4915a67ba3ec267f9691f2: Status 404 returned error can't find the container with id 8f1471d5aa8cdb387116c77d6dc9ce1d0d5938138d4915a67ba3ec267f9691f2 Oct 30 18:38:33.731817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:33.731796 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7"] Oct 30 18:38:34.214374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:34.214278 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" event={"ID":"a52b900a-f62c-45e5-bc7b-b3bb1afb7075","Type":"ContainerStarted","Data":"43f2b83a93ee874839e7d99cdb33965bc82f9976167dabd65f2706618b9e104f"} Oct 30 18:38:34.214374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:34.214345 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" event={"ID":"a52b900a-f62c-45e5-bc7b-b3bb1afb7075","Type":"ContainerStarted","Data":"8f1471d5aa8cdb387116c77d6dc9ce1d0d5938138d4915a67ba3ec267f9691f2"} Oct 30 18:38:34.216208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:34.216168 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" event={"ID":"71abc140-6296-4993-b427-61a9977cabde","Type":"ContainerStarted","Data":"f2fda8eb22cb8125cf285777fa2b36b2edee1a627491b39f1f6376c4d2814852"} Oct 30 18:38:34.216341 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:34.216213 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" event={"ID":"71abc140-6296-4993-b427-61a9977cabde","Type":"ContainerStarted","Data":"918be1f80f32ad3c551e891b59311513296e2eee8035c5444e24310481b03e64"} Oct 30 18:38:34.264068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:34.264016 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" podStartSLOduration=1.264002506 podStartE2EDuration="1.264002506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:34.262738445 +0000 UTC m=+2773.219742291" watchObservedRunningTime="2026-10-30 18:38:34.264002506 +0000 UTC m=+2773.221006338" Oct 30 18:38:34.336001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:34.335930 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" podStartSLOduration=2.335914843 podStartE2EDuration="2.335914843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:34.332111592 +0000 UTC m=+2773.289115427" watchObservedRunningTime="2026-10-30 18:38:34.335914843 +0000 UTC m=+2773.292918679" Oct 30 18:38:35.079297 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:38:35.079241 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:38:35.224015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:35.223977 4883 generic.go:356] "Generic (PLEG): container finished" podID="71abc140-6296-4993-b427-61a9977cabde" containerID="f2fda8eb22cb8125cf285777fa2b36b2edee1a627491b39f1f6376c4d2814852" exitCode=0 Oct 30 18:38:35.224437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:35.224065 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" event={"ID":"71abc140-6296-4993-b427-61a9977cabde","Type":"ContainerDied","Data":"f2fda8eb22cb8125cf285777fa2b36b2edee1a627491b39f1f6376c4d2814852"} Oct 30 18:38:35.225721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:35.225692 4883 generic.go:356] "Generic (PLEG): container finished" podID="a52b900a-f62c-45e5-bc7b-b3bb1afb7075" containerID="43f2b83a93ee874839e7d99cdb33965bc82f9976167dabd65f2706618b9e104f" exitCode=0 Oct 30 18:38:35.225842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:35.225755 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" event={"ID":"a52b900a-f62c-45e5-bc7b-b3bb1afb7075","Type":"ContainerDied","Data":"43f2b83a93ee874839e7d99cdb33965bc82f9976167dabd65f2706618b9e104f"} Oct 30 18:38:36.457104 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.457064 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:36.491833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.491763 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:36.569215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.569165 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71abc140-6296-4993-b427-61a9977cabde-kube-api-access-gkmh6\" (UniqueName: \"kubernetes.io/projected/71abc140-6296-4993-b427-61a9977cabde-kube-api-access-gkmh6\") pod \"71abc140-6296-4993-b427-61a9977cabde\" (UID: \"71abc140-6296-4993-b427-61a9977cabde\") " Oct 30 18:38:36.569386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.569314 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71abc140-6296-4993-b427-61a9977cabde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71abc140-6296-4993-b427-61a9977cabde-maas-api-openshift-ca\") pod \"71abc140-6296-4993-b427-61a9977cabde\" (UID: \"71abc140-6296-4993-b427-61a9977cabde\") " Oct 30 18:38:36.569436 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.569406 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71abc140-6296-4993-b427-61a9977cabde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71abc140-6296-4993-b427-61a9977cabde-maas-api-xks-ca\") pod \"71abc140-6296-4993-b427-61a9977cabde\" (UID: \"71abc140-6296-4993-b427-61a9977cabde\") " Oct 30 18:38:36.571167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.571108 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71abc140-6296-4993-b427-61a9977cabde-maas-api-openshift-ca" pod "71abc140-6296-4993-b427-61a9977cabde" (UID: "71abc140-6296-4993-b427-61a9977cabde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:36.583324 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.583267 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71abc140-6296-4993-b427-61a9977cabde-kube-api-access-gkmh6" pod "71abc140-6296-4993-b427-61a9977cabde" (UID: "71abc140-6296-4993-b427-61a9977cabde"). InnerVolumeSpecName "kube-api-access-gkmh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:36.584528 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.584499 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkmh6\" (UniqueName: \"kubernetes.io/projected/71abc140-6296-4993-b427-61a9977cabde-kube-api-access-gkmh6\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:36.585987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.585371 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71abc140-6296-4993-b427-61a9977cabde-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:36.597802 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.597743 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9"] Oct 30 18:38:36.616035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.607465 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71abc140-6296-4993-b427-61a9977cabde-maas-api-xks-ca" pod "71abc140-6296-4993-b427-61a9977cabde" (UID: "71abc140-6296-4993-b427-61a9977cabde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:36.616035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.611507 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-57ee15-jqtz9"] Oct 30 18:38:36.685820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.685785 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-openshift-ca\") pod \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\" (UID: \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\") " Oct 30 18:38:36.686034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.685892 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-xks-ca\") pod \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\" (UID: \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\") " Oct 30 18:38:36.686094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.686053 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-kube-api-access-jx9gs\" (UniqueName: \"kubernetes.io/projected/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-kube-api-access-jx9gs\") pod \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\" (UID: \"a52b900a-f62c-45e5-bc7b-b3bb1afb7075\") " Oct 30 18:38:36.686233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.686209 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-openshift-ca" pod "a52b900a-f62c-45e5-bc7b-b3bb1afb7075" (UID: "a52b900a-f62c-45e5-bc7b-b3bb1afb7075"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:36.686565 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.686543 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:36.686675 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.686568 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71abc140-6296-4993-b427-61a9977cabde-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:36.688122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.688099 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-xks-ca" pod "a52b900a-f62c-45e5-bc7b-b3bb1afb7075" (UID: "a52b900a-f62c-45e5-bc7b-b3bb1afb7075"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:36.688122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.688109 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-kube-api-access-jx9gs" pod "a52b900a-f62c-45e5-bc7b-b3bb1afb7075" (UID: "a52b900a-f62c-45e5-bc7b-b3bb1afb7075"). InnerVolumeSpecName "kube-api-access-jx9gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:36.787826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.787745 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:36.787826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:36.787777 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx9gs\" (UniqueName: \"kubernetes.io/projected/a52b900a-f62c-45e5-bc7b-b3bb1afb7075-kube-api-access-jx9gs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:37.180794 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:37.180750 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad413fc7-2b98-4faf-af7d-285b1f4bc308" path="/var/lib/kubelet/pods/ad413fc7-2b98-4faf-af7d-285b1f4bc308/volumes" Oct 30 18:38:37.244903 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:37.244872 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" Oct 30 18:38:37.245120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:37.244901 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5" event={"ID":"71abc140-6296-4993-b427-61a9977cabde","Type":"ContainerDied","Data":"918be1f80f32ad3c551e891b59311513296e2eee8035c5444e24310481b03e64"} Oct 30 18:38:37.245120 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:37.244966 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="918be1f80f32ad3c551e891b59311513296e2eee8035c5444e24310481b03e64" Oct 30 18:38:37.246551 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:37.246528 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" event={"ID":"a52b900a-f62c-45e5-bc7b-b3bb1afb7075","Type":"ContainerDied","Data":"8f1471d5aa8cdb387116c77d6dc9ce1d0d5938138d4915a67ba3ec267f9691f2"} Oct 30 18:38:37.246698 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:37.246565 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7" Oct 30 18:38:37.246698 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:37.246566 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f1471d5aa8cdb387116c77d6dc9ce1d0d5938138d4915a67ba3ec267f9691f2" Oct 30 18:38:41.668812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:41.668776 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj"] Oct 30 18:38:41.681725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:41.681682 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6e8fca-dgsjj"] Oct 30 18:38:43.179359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:43.179326 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c51dbdd-fc11-42b7-8918-3b957502118c" path="/var/lib/kubelet/pods/9c51dbdd-fc11-42b7-8918-3b957502118c/volumes" Oct 30 18:38:47.150227 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.150165 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerName="main" containerID="cri-o://0a1f22cef49d5861ec564ee9cf925668fdf2320069e6c61d64faba4ef427a4f2" gracePeriod=45 Oct 30 18:38:47.412414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.412324 4883 generic.go:356] "Generic (PLEG): container finished" podID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerID="0a1f22cef49d5861ec564ee9cf925668fdf2320069e6c61d64faba4ef427a4f2" exitCode=0 Oct 30 18:38:47.412569 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.412399 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" event={"ID":"df3fde75-25a2-4fde-b7a4-78e412c6c696","Type":"ContainerDied","Data":"0a1f22cef49d5861ec564ee9cf925668fdf2320069e6c61d64faba4ef427a4f2"} Oct 30 18:38:47.448353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.448325 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:38:47.567300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.566697 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df3fde75-25a2-4fde-b7a4-78e412c6c696-kube-api-access-jftts\" (UniqueName: \"kubernetes.io/projected/df3fde75-25a2-4fde-b7a4-78e412c6c696-kube-api-access-jftts\") pod \"df3fde75-25a2-4fde-b7a4-78e412c6c696\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " Oct 30 18:38:47.567300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.566804 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-dshm\") pod \"df3fde75-25a2-4fde-b7a4-78e412c6c696\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " Oct 30 18:38:47.567300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.567024 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df3fde75-25a2-4fde-b7a4-78e412c6c696-tls-certs\" (UniqueName: \"kubernetes.io/secret/df3fde75-25a2-4fde-b7a4-78e412c6c696-tls-certs\") pod \"df3fde75-25a2-4fde-b7a4-78e412c6c696\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " Oct 30 18:38:47.567300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.567084 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-tmp-dir\") pod \"df3fde75-25a2-4fde-b7a4-78e412c6c696\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " Oct 30 18:38:47.567300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.567124 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-model-cache\") pod \"df3fde75-25a2-4fde-b7a4-78e412c6c696\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " Oct 30 18:38:47.567300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.567191 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-home\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-home\") pod \"df3fde75-25a2-4fde-b7a4-78e412c6c696\" (UID: \"df3fde75-25a2-4fde-b7a4-78e412c6c696\") " Oct 30 18:38:47.568185 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.568157 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-home" pod "df3fde75-25a2-4fde-b7a4-78e412c6c696" (UID: "df3fde75-25a2-4fde-b7a4-78e412c6c696"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:47.570655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.570621 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3fde75-25a2-4fde-b7a4-78e412c6c696-kube-api-access-jftts" pod "df3fde75-25a2-4fde-b7a4-78e412c6c696" (UID: "df3fde75-25a2-4fde-b7a4-78e412c6c696"). InnerVolumeSpecName "kube-api-access-jftts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:47.571512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.571371 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-tmp-dir" pod "df3fde75-25a2-4fde-b7a4-78e412c6c696" (UID: "df3fde75-25a2-4fde-b7a4-78e412c6c696"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:47.571727 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.571662 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-model-cache" pod "df3fde75-25a2-4fde-b7a4-78e412c6c696" (UID: "df3fde75-25a2-4fde-b7a4-78e412c6c696"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:47.574651 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.574582 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-dshm" pod "df3fde75-25a2-4fde-b7a4-78e412c6c696" (UID: "df3fde75-25a2-4fde-b7a4-78e412c6c696"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:47.575139 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.575115 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df3fde75-25a2-4fde-b7a4-78e412c6c696-tls-certs" pod "df3fde75-25a2-4fde-b7a4-78e412c6c696" (UID: "df3fde75-25a2-4fde-b7a4-78e412c6c696"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:47.669622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.669532 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jftts\" (UniqueName: \"kubernetes.io/projected/df3fde75-25a2-4fde-b7a4-78e412c6c696-kube-api-access-jftts\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:47.669622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.669569 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:47.669622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.669584 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3fde75-25a2-4fde-b7a4-78e412c6c696-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:47.669622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.669598 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:47.669622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.669607 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:47.669622 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.669616 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3fde75-25a2-4fde-b7a4-78e412c6c696-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:47.685049 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.684989 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" podUID="85b87546-1283-431a-b62a-2718b90bc3cb" containerName="main" containerID="cri-o://5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d" gracePeriod=45 Oct 30 18:38:47.960964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:47.960919 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:48.074894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.074853 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-model-cache\") pod \"85b87546-1283-431a-b62a-2718b90bc3cb\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " Oct 30 18:38:48.075123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.075016 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-home\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-home\") pod \"85b87546-1283-431a-b62a-2718b90bc3cb\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " Oct 30 18:38:48.075123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.075084 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b87546-1283-431a-b62a-2718b90bc3cb-kube-api-access-4wvll\" (UniqueName: \"kubernetes.io/projected/85b87546-1283-431a-b62a-2718b90bc3cb-kube-api-access-4wvll\") pod \"85b87546-1283-431a-b62a-2718b90bc3cb\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " Oct 30 18:38:48.075261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.075133 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-tmp-dir\") pod \"85b87546-1283-431a-b62a-2718b90bc3cb\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " Oct 30 18:38:48.075261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.075192 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85b87546-1283-431a-b62a-2718b90bc3cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/85b87546-1283-431a-b62a-2718b90bc3cb-tls-certs\") pod \"85b87546-1283-431a-b62a-2718b90bc3cb\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " Oct 30 18:38:48.075261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.075190 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-model-cache" pod "85b87546-1283-431a-b62a-2718b90bc3cb" (UID: "85b87546-1283-431a-b62a-2718b90bc3cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:48.075261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.075229 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-home" pod "85b87546-1283-431a-b62a-2718b90bc3cb" (UID: "85b87546-1283-431a-b62a-2718b90bc3cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:48.075477 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.075445 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-dshm\") pod \"85b87546-1283-431a-b62a-2718b90bc3cb\" (UID: \"85b87546-1283-431a-b62a-2718b90bc3cb\") " Oct 30 18:38:48.075814 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.075780 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-tmp-dir" pod "85b87546-1283-431a-b62a-2718b90bc3cb" (UID: "85b87546-1283-431a-b62a-2718b90bc3cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:48.076277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.076245 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:48.076277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.076272 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:48.076454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.076287 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:48.077713 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.077692 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b87546-1283-431a-b62a-2718b90bc3cb-kube-api-access-4wvll" pod "85b87546-1283-431a-b62a-2718b90bc3cb" (UID: "85b87546-1283-431a-b62a-2718b90bc3cb"). InnerVolumeSpecName "kube-api-access-4wvll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:48.077816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.077738 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85b87546-1283-431a-b62a-2718b90bc3cb-tls-certs" pod "85b87546-1283-431a-b62a-2718b90bc3cb" (UID: "85b87546-1283-431a-b62a-2718b90bc3cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:48.077976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.077924 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-dshm" pod "85b87546-1283-431a-b62a-2718b90bc3cb" (UID: "85b87546-1283-431a-b62a-2718b90bc3cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:48.177925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.177897 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b87546-1283-431a-b62a-2718b90bc3cb-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:48.177925 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.177924 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b87546-1283-431a-b62a-2718b90bc3cb-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:48.178313 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.177946 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wvll\" (UniqueName: \"kubernetes.io/projected/85b87546-1283-431a-b62a-2718b90bc3cb-kube-api-access-4wvll\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:48.429580 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.429545 4883 generic.go:356] "Generic (PLEG): container finished" podID="85b87546-1283-431a-b62a-2718b90bc3cb" containerID="5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d" exitCode=0 Oct 30 18:38:48.429776 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.429612 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" Oct 30 18:38:48.429776 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.429626 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" event={"ID":"85b87546-1283-431a-b62a-2718b90bc3cb","Type":"ContainerDied","Data":"5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d"} Oct 30 18:38:48.429776 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.429678 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8" event={"ID":"85b87546-1283-431a-b62a-2718b90bc3cb","Type":"ContainerDied","Data":"3826c2f031cbf569181e595b10c78eed9b1dc942baf43f56f9a037ae518ebced"} Oct 30 18:38:48.429776 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.429702 4883 scope.go:122] "RemoveContainer" containerID="5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d" Oct 30 18:38:48.431473 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.431439 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" event={"ID":"df3fde75-25a2-4fde-b7a4-78e412c6c696","Type":"ContainerDied","Data":"8e345750960b6c75ae308134b2ff422209dbcc2574414f7c44e067764f0e301e"} Oct 30 18:38:48.431587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.431545 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j" Oct 30 18:38:48.441162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.441128 4883 scope.go:122] "RemoveContainer" containerID="5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d" Oct 30 18:38:48.441427 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:38:48.441395 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d\": container with ID starting with 5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d not found: ID does not exist" containerID="5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d" Oct 30 18:38:48.441512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.441442 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d"} err="failed to get container status \"5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d\": rpc error: code = NotFound desc = could not find container \"5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d\": container with ID starting with 5fec7867b4af5dc5283c8524932ddb459ef48e963265df30e86190e05311163d not found: ID does not exist" Oct 30 18:38:48.441512 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.441463 4883 scope.go:122] "RemoveContainer" containerID="0a1f22cef49d5861ec564ee9cf925668fdf2320069e6c61d64faba4ef427a4f2" Oct 30 18:38:48.524308 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.524271 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8"] Oct 30 18:38:48.540009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.539965 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-a1a31f/explicit-model-a1a31f-kserve-59474bc57-lwvf8"] Oct 30 18:38:48.582498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.582465 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j"] Oct 30 18:38:48.591816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.591779 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-58d08b/autores-model-58d08b-kserve-7768d857f7-fln5j"] Oct 30 18:38:48.962771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.962729 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr"] Oct 30 18:38:48.963141 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.963091 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" containerID="cri-o://e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc" gracePeriod=30 Oct 30 18:38:48.986377 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.986335 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2"] Oct 30 18:38:48.986564 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:48.986530 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" containerID="cri-o://f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956" gracePeriod=30 Oct 30 18:38:49.043241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:49.023081 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx"] Oct 30 18:38:49.043241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:49.023350 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" containerID="cri-o://33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226" gracePeriod=30 Oct 30 18:38:49.214659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:49.214573 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b87546-1283-431a-b62a-2718b90bc3cb" path="/var/lib/kubelet/pods/85b87546-1283-431a-b62a-2718b90bc3cb/volumes" Oct 30 18:38:49.215205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:49.215183 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" path="/var/lib/kubelet/pods/df3fde75-25a2-4fde-b7a4-78e412c6c696/volumes" Oct 30 18:38:51.813289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:51.813245 4883 patch_prober.go:29] interesting pod/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:38:51.813669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:51.813313 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:38:52.015813 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:52.012878 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r"] Oct 30 18:38:52.015813 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:52.013293 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" podUID="0086e9f0-1201-4263-b710-5417225349ee" containerName="istio-proxy" containerID="cri-o://9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568" gracePeriod=30 Oct 30 18:38:52.035572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:52.035101 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj"] Oct 30 18:38:52.035572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:52.035361 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" containerID="cri-o://c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c" gracePeriod=30 Oct 30 18:38:52.040162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:52.040129 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8"] Oct 30 18:38:52.040405 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:52.040379 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" containerID="cri-o://8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218" gracePeriod=30 Oct 30 18:38:52.731284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:52.731244 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7"] Oct 30 18:38:52.736273 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:52.736245 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a1a31f-djxk7"] Oct 30 18:38:53.178766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:53.178724 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a52b900a-f62c-45e5-bc7b-b3bb1afb7075" path="/var/lib/kubelet/pods/a52b900a-f62c-45e5-bc7b-b3bb1afb7075/volumes" Oct 30 18:38:54.347903 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.347866 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:38:54.364478 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.364450 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:38:54.397249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.397218 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:38:54.497356 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.497266 4883 generic.go:356] "Generic (PLEG): container finished" podID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerID="33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226" exitCode=0 Oct 30 18:38:54.497356 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.497344 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" Oct 30 18:38:54.497578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.497362 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" event={"ID":"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9","Type":"ContainerDied","Data":"33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226"} Oct 30 18:38:54.497578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.497451 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx" event={"ID":"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9","Type":"ContainerDied","Data":"c827f282f8daa7a00b2340a016383cd2511c2bdff2f5dc04ec2a87a2d5c6fbdf"} Oct 30 18:38:54.497578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.497499 4883 scope.go:122] "RemoveContainer" containerID="33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226" Oct 30 18:38:54.498627 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.498600 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-credential-socket\") pod \"f847895f-c06c-4e79-8f10-9305e980856a\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " Oct 30 18:38:54.498797 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.498646 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-envoy\") pod \"f847895f-c06c-4e79-8f10-9305e980856a\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " Oct 30 18:38:54.498797 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.498694 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istiod-ca-cert\") pod \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " Oct 30 18:38:54.498797 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.498728 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c5b4960-719d-4f28-893f-a6d4a7e41369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c5b4960-719d-4f28-893f-a6d4a7e41369-istiod-ca-cert\") pod \"3c5b4960-719d-4f28-893f-a6d4a7e41369\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " Oct 30 18:38:54.498797 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.498774 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-token\" (UniqueName: \"kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-token\") pod \"3c5b4960-719d-4f28-893f-a6d4a7e41369\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " Oct 30 18:38:54.499165 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.498994 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-credential-socket" pod "f847895f-c06c-4e79-8f10-9305e980856a" (UID: "f847895f-c06c-4e79-8f10-9305e980856a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.499165 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499137 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istiod-ca-cert" pod "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" (UID: "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.499287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499186 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-data\") pod \"f847895f-c06c-4e79-8f10-9305e980856a\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " Oct 30 18:38:54.499287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499214 4883 generic.go:356] "Generic (PLEG): container finished" podID="f847895f-c06c-4e79-8f10-9305e980856a" containerID="e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc" exitCode=0 Oct 30 18:38:54.499287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499253 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-data\") pod \"3c5b4960-719d-4f28-893f-a6d4a7e41369\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " Oct 30 18:38:54.499434 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499289 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-kube-api-access-8z2d5\" (UniqueName: \"kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-kube-api-access-8z2d5\") pod \"f847895f-c06c-4e79-8f10-9305e980856a\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " Oct 30 18:38:54.499434 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499389 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" event={"ID":"f847895f-c06c-4e79-8f10-9305e980856a","Type":"ContainerDied","Data":"e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc"} Oct 30 18:38:54.499434 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499408 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-podinfo\") pod \"3c5b4960-719d-4f28-893f-a6d4a7e41369\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " Oct 30 18:38:54.499434 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499424 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" Oct 30 18:38:54.499661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499435 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr" event={"ID":"f847895f-c06c-4e79-8f10-9305e980856a","Type":"ContainerDied","Data":"e64c286edb1bc49ff69c8b8f51ae9ce7580991f06e648908eff332d9f9b70644"} Oct 30 18:38:54.499661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499440 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-certs\") pod \"3c5b4960-719d-4f28-893f-a6d4a7e41369\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " Oct 30 18:38:54.499661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499478 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-socket\") pod \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " Oct 30 18:38:54.499661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499513 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f847895f-c06c-4e79-8f10-9305e980856a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f847895f-c06c-4e79-8f10-9305e980856a-istiod-ca-cert\") pod \"f847895f-c06c-4e79-8f10-9305e980856a\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " Oct 30 18:38:54.499661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499542 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-envoy\") pod \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " Oct 30 18:38:54.499661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499591 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f847895f-c06c-4e79-8f10-9305e980856a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f847895f-c06c-4e79-8f10-9305e980856a-istio-podinfo\") pod \"f847895f-c06c-4e79-8f10-9305e980856a\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " Oct 30 18:38:54.499661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499628 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-data" pod "3c5b4960-719d-4f28-893f-a6d4a7e41369" (UID: "3c5b4960-719d-4f28-893f-a6d4a7e41369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.499661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499634 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-socket\") pod \"f847895f-c06c-4e79-8f10-9305e980856a\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " Oct 30 18:38:54.500110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499695 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-data\") pod \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " Oct 30 18:38:54.500110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499735 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-token\" (UniqueName: \"kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-token\") pod \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " Oct 30 18:38:54.500110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499803 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-socket\") pod \"3c5b4960-719d-4f28-893f-a6d4a7e41369\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " Oct 30 18:38:54.500110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499853 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-kube-api-access-4wncd\" (UniqueName: \"kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-kube-api-access-4wncd\") pod \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " Oct 30 18:38:54.500110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499898 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-envoy\") pod \"3c5b4960-719d-4f28-893f-a6d4a7e41369\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " Oct 30 18:38:54.500110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499953 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-certs\") pod \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " Oct 30 18:38:54.500110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499959 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-data" pod "f847895f-c06c-4e79-8f10-9305e980856a" (UID: "f847895f-c06c-4e79-8f10-9305e980856a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.500110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.499988 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-certs\") pod \"f847895f-c06c-4e79-8f10-9305e980856a\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " Oct 30 18:38:54.500110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.500016 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-credential-socket\") pod \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " Oct 30 18:38:54.500110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.500032 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5b4960-719d-4f28-893f-a6d4a7e41369-istiod-ca-cert" pod "3c5b4960-719d-4f28-893f-a6d4a7e41369" (UID: "3c5b4960-719d-4f28-893f-a6d4a7e41369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.500110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.500071 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-istio-token\" (UniqueName: \"kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-istio-token\") pod \"f847895f-c06c-4e79-8f10-9305e980856a\" (UID: \"f847895f-c06c-4e79-8f10-9305e980856a\") " Oct 30 18:38:54.500110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.500109 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-podinfo\") pod \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\" (UID: \"c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9\") " Oct 30 18:38:54.500695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.500223 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-kube-api-access-dgrq5\" (UniqueName: \"kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-kube-api-access-dgrq5\") pod \"3c5b4960-719d-4f28-893f-a6d4a7e41369\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " Oct 30 18:38:54.500695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.500273 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-credential-socket\") pod \"3c5b4960-719d-4f28-893f-a6d4a7e41369\" (UID: \"3c5b4960-719d-4f28-893f-a6d4a7e41369\") " Oct 30 18:38:54.500695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.500398 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-certs" pod "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" (UID: "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.500869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.500680 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-certs" pod "3c5b4960-719d-4f28-893f-a6d4a7e41369" (UID: "3c5b4960-719d-4f28-893f-a6d4a7e41369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.501015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.500992 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-socket" pod "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" (UID: "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.501113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.500999 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-credential-socket" pod "3c5b4960-719d-4f28-893f-a6d4a7e41369" (UID: "3c5b4960-719d-4f28-893f-a6d4a7e41369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.501202 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.501184 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-credential-socket" pod "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" (UID: "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.501665 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.501627 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-socket" pod "3c5b4960-719d-4f28-893f-a6d4a7e41369" (UID: "3c5b4960-719d-4f28-893f-a6d4a7e41369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.502765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.501953 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-certs" pod "f847895f-c06c-4e79-8f10-9305e980856a" (UID: "f847895f-c06c-4e79-8f10-9305e980856a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.502765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.502106 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f847895f-c06c-4e79-8f10-9305e980856a-istiod-ca-cert" pod "f847895f-c06c-4e79-8f10-9305e980856a" (UID: "f847895f-c06c-4e79-8f10-9305e980856a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.502765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.502350 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-token" pod "3c5b4960-719d-4f28-893f-a6d4a7e41369" (UID: "3c5b4960-719d-4f28-893f-a6d4a7e41369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.502765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.502400 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-data" pod "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" (UID: "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.502765 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.502623 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-socket" pod "f847895f-c06c-4e79-8f10-9305e980856a" (UID: "f847895f-c06c-4e79-8f10-9305e980856a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.502879 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.502903 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.502919 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.502951 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c5b4960-719d-4f28-893f-a6d4a7e41369-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.502965 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.502978 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.502991 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503004 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503016 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503030 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f847895f-c06c-4e79-8f10-9305e980856a-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503043 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503056 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503071 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503084 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.503099 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503097 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.504279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503111 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.504279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503696 4883 generic.go:356] "Generic (PLEG): container finished" podID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerID="f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956" exitCode=0 Oct 30 18:38:54.504279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503752 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" Oct 30 18:38:54.504279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503756 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" event={"ID":"3c5b4960-719d-4f28-893f-a6d4a7e41369","Type":"ContainerDied","Data":"f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956"} Oct 30 18:38:54.504279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.503866 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2" event={"ID":"3c5b4960-719d-4f28-893f-a6d4a7e41369","Type":"ContainerDied","Data":"a696da2c73885292e6a36b92573a8469529615885d82169690aa7291d16bcca1"} Oct 30 18:38:54.504574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.504390 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-token" pod "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" (UID: "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.508064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.508015 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-podinfo" pod "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" (UID: "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:54.508064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.508051 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f847895f-c06c-4e79-8f10-9305e980856a-istio-podinfo" pod "f847895f-c06c-4e79-8f10-9305e980856a" (UID: "f847895f-c06c-4e79-8f10-9305e980856a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:54.508240 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.508072 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-envoy" pod "3c5b4960-719d-4f28-893f-a6d4a7e41369" (UID: "3c5b4960-719d-4f28-893f-a6d4a7e41369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.508240 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.508080 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-istio-token" pod "f847895f-c06c-4e79-8f10-9305e980856a" (UID: "f847895f-c06c-4e79-8f10-9305e980856a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.508240 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.508138 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-kube-api-access-4wncd" pod "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" (UID: "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9"). InnerVolumeSpecName "kube-api-access-4wncd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.508240 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.508183 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-podinfo" pod "3c5b4960-719d-4f28-893f-a6d4a7e41369" (UID: "3c5b4960-719d-4f28-893f-a6d4a7e41369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:54.508430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.508272 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-envoy" pod "f847895f-c06c-4e79-8f10-9305e980856a" (UID: "f847895f-c06c-4e79-8f10-9305e980856a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.508430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.508292 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-kube-api-access-8z2d5" pod "f847895f-c06c-4e79-8f10-9305e980856a" (UID: "f847895f-c06c-4e79-8f10-9305e980856a"). InnerVolumeSpecName "kube-api-access-8z2d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.509621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.509597 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-kube-api-access-dgrq5" pod "3c5b4960-719d-4f28-893f-a6d4a7e41369" (UID: "3c5b4960-719d-4f28-893f-a6d4a7e41369"). InnerVolumeSpecName "kube-api-access-dgrq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.511093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.511069 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-envoy" pod "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" (UID: "c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:54.513322 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.513301 4883 scope.go:122] "RemoveContainer" containerID="33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226" Oct 30 18:38:54.513678 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:38:54.513658 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226\": container with ID starting with 33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226 not found: ID does not exist" containerID="33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226" Oct 30 18:38:54.513777 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.513688 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226"} err="failed to get container status \"33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226\": rpc error: code = NotFound desc = could not find container \"33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226\": container with ID starting with 33df801beb2aba9833b6e8417caac39309e77c4ed94a3ccd0626590eac2b3226 not found: ID does not exist" Oct 30 18:38:54.513777 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.513707 4883 scope.go:122] "RemoveContainer" containerID="e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc" Oct 30 18:38:54.540542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.539616 4883 scope.go:122] "RemoveContainer" containerID="e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc" Oct 30 18:38:54.540542 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:38:54.540160 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc\": container with ID starting with e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc not found: ID does not exist" containerID="e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc" Oct 30 18:38:54.540542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.540203 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc"} err="failed to get container status \"e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc\": rpc error: code = NotFound desc = could not find container \"e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc\": container with ID starting with e757614afaba822555d58de5bcfcf75545ae298a3f1c59395d3d7515f1fed4fc not found: ID does not exist" Oct 30 18:38:54.540542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.540231 4883 scope.go:122] "RemoveContainer" containerID="f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956" Oct 30 18:38:54.553812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.553280 4883 scope.go:122] "RemoveContainer" containerID="f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956" Oct 30 18:38:54.554124 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:38:54.554050 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956\": container with ID starting with f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956 not found: ID does not exist" containerID="f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956" Oct 30 18:38:54.554124 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.554090 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956"} err="failed to get container status \"f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956\": rpc error: code = NotFound desc = could not find container \"f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956\": container with ID starting with f3e1a079445cfc2abb9652fcf6fe111370eb89886608884ded4a829500d5e956 not found: ID does not exist" Oct 30 18:38:54.604982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.604928 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f847895f-c06c-4e79-8f10-9305e980856a-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.604982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.604976 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z2d5\" (UniqueName: \"kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-kube-api-access-8z2d5\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.605245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.605019 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.605245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.605038 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.605245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.605055 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f847895f-c06c-4e79-8f10-9305e980856a-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.605245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.605070 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.605245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.605081 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wncd\" (UniqueName: \"kubernetes.io/projected/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-kube-api-access-4wncd\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.605245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.605095 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c5b4960-719d-4f28-893f-a6d4a7e41369-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.605245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.605108 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f847895f-c06c-4e79-8f10-9305e980856a-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.605245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.605121 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.605245 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.605133 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgrq5\" (UniqueName: \"kubernetes.io/projected/3c5b4960-719d-4f28-893f-a6d4a7e41369-kube-api-access-dgrq5\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.864442 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.864407 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx"] Oct 30 18:38:54.869994 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.869960 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-6bvtx"] Oct 30 18:38:54.950394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.949116 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr"] Oct 30 18:38:54.954888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.954848 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-2xtvr"] Oct 30 18:38:54.984237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.984203 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2"] Oct 30 18:38:54.990411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:54.990379 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-57ee15-openshift-default-b4df69477-xkxh2"] Oct 30 18:38:55.178514 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:55.178474 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" path="/var/lib/kubelet/pods/3c5b4960-719d-4f28-893f-a6d4a7e41369/volumes" Oct 30 18:38:55.179095 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:55.179078 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" path="/var/lib/kubelet/pods/c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9/volumes" Oct 30 18:38:55.179644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:55.179629 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f847895f-c06c-4e79-8f10-9305e980856a" path="/var/lib/kubelet/pods/f847895f-c06c-4e79-8f10-9305e980856a/volumes" Oct 30 18:38:57.402001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.401967 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:38:57.458468 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.458444 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:38:57.461591 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.461570 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:38:57.529627 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.529592 4883 generic.go:356] "Generic (PLEG): container finished" podID="30b7c407-135e-45ee-96b6-b09469351896" containerID="c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c" exitCode=0 Oct 30 18:38:57.529811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.529646 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" Oct 30 18:38:57.529811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.529676 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" event={"ID":"30b7c407-135e-45ee-96b6-b09469351896","Type":"ContainerDied","Data":"c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c"} Oct 30 18:38:57.529811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.529744 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj" event={"ID":"30b7c407-135e-45ee-96b6-b09469351896","Type":"ContainerDied","Data":"39ae3617f251e6ea1f76f1054ce9d085f2380c8e4d276b7745753d4271d7c804"} Oct 30 18:38:57.529811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.529778 4883 scope.go:122] "RemoveContainer" containerID="c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c" Oct 30 18:38:57.531600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.531575 4883 generic.go:356] "Generic (PLEG): container finished" podID="0086e9f0-1201-4263-b710-5417225349ee" containerID="9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568" exitCode=0 Oct 30 18:38:57.531741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.531624 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" Oct 30 18:38:57.531741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.531678 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" event={"ID":"0086e9f0-1201-4263-b710-5417225349ee","Type":"ContainerDied","Data":"9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568"} Oct 30 18:38:57.531869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.531742 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r" event={"ID":"0086e9f0-1201-4263-b710-5417225349ee","Type":"ContainerDied","Data":"84a1229ab50f804fc95120f3cbaff303a88992bab900e1ee9393dd718c8655af"} Oct 30 18:38:57.533360 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.533334 4883 generic.go:356] "Generic (PLEG): container finished" podID="60201992-7051-4dbc-8798-ef7ae9152518" containerID="8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218" exitCode=0 Oct 30 18:38:57.533501 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.533386 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" Oct 30 18:38:57.533501 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.533414 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" event={"ID":"60201992-7051-4dbc-8798-ef7ae9152518","Type":"ContainerDied","Data":"8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218"} Oct 30 18:38:57.533501 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.533467 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8" event={"ID":"60201992-7051-4dbc-8798-ef7ae9152518","Type":"ContainerDied","Data":"0c396a28c223dee92f3657cdbe02a1e31ef52cd7b1269702e4ca706f4ec27cc4"} Oct 30 18:38:57.543995 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.543962 4883 scope.go:122] "RemoveContainer" containerID="c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c" Oct 30 18:38:57.544603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544285 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0086e9f0-1201-4263-b710-5417225349ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0086e9f0-1201-4263-b710-5417225349ee-istio-podinfo\") pod \"0086e9f0-1201-4263-b710-5417225349ee\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " Oct 30 18:38:57.544603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544350 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60201992-7051-4dbc-8798-ef7ae9152518-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60201992-7051-4dbc-8798-ef7ae9152518-istio-podinfo\") pod \"60201992-7051-4dbc-8798-ef7ae9152518\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " Oct 30 18:38:57.544603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544386 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-envoy\") pod \"60201992-7051-4dbc-8798-ef7ae9152518\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " Oct 30 18:38:57.544603 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:38:57.544388 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c\": container with ID starting with c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c not found: ID does not exist" containerID="c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c" Oct 30 18:38:57.544603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544416 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-certs\") pod \"0086e9f0-1201-4263-b710-5417225349ee\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " Oct 30 18:38:57.544603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544430 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c"} err="failed to get container status \"c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c\": rpc error: code = NotFound desc = could not find container \"c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c\": container with ID starting with c077dd97346d0a49831fe04a7ace5a212b3ce0aa9f52719e6f6598a207ec408c not found: ID does not exist" Oct 30 18:38:57.544603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544455 4883 scope.go:122] "RemoveContainer" containerID="9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568" Oct 30 18:38:57.544603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544482 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-kube-api-access-vpdhs\" (UniqueName: \"kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-kube-api-access-vpdhs\") pod \"0086e9f0-1201-4263-b710-5417225349ee\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " Oct 30 18:38:57.544603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544517 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-istio-token\" (UniqueName: \"kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-istio-token\") pod \"30b7c407-135e-45ee-96b6-b09469351896\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " Oct 30 18:38:57.544603 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544585 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-credential-socket\") pod \"60201992-7051-4dbc-8798-ef7ae9152518\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " Oct 30 18:38:57.545279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544637 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-socket\") pod \"0086e9f0-1201-4263-b710-5417225349ee\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " Oct 30 18:38:57.545279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544670 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b7c407-135e-45ee-96b6-b09469351896-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b7c407-135e-45ee-96b6-b09469351896-istiod-ca-cert\") pod \"30b7c407-135e-45ee-96b6-b09469351896\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " Oct 30 18:38:57.545279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544707 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-envoy\") pod \"0086e9f0-1201-4263-b710-5417225349ee\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " Oct 30 18:38:57.545279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544738 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-data\") pod \"30b7c407-135e-45ee-96b6-b09469351896\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " Oct 30 18:38:57.545279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544754 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-certs" pod "0086e9f0-1201-4263-b710-5417225349ee" (UID: "0086e9f0-1201-4263-b710-5417225349ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.545279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544783 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-credential-socket\") pod \"0086e9f0-1201-4263-b710-5417225349ee\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " Oct 30 18:38:57.545279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544829 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-credential-socket\") pod \"30b7c407-135e-45ee-96b6-b09469351896\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " Oct 30 18:38:57.545279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.544878 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30b7c407-135e-45ee-96b6-b09469351896-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b7c407-135e-45ee-96b6-b09469351896-istio-podinfo\") pod \"30b7c407-135e-45ee-96b6-b09469351896\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " Oct 30 18:38:57.545742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545298 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-socket" pod "0086e9f0-1201-4263-b710-5417225349ee" (UID: "0086e9f0-1201-4263-b710-5417225349ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.545742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545313 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-credential-socket" pod "0086e9f0-1201-4263-b710-5417225349ee" (UID: "0086e9f0-1201-4263-b710-5417225349ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.545742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545336 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b7c407-135e-45ee-96b6-b09469351896-istiod-ca-cert" pod "30b7c407-135e-45ee-96b6-b09469351896" (UID: "30b7c407-135e-45ee-96b6-b09469351896"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:57.545742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545394 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-data\") pod \"60201992-7051-4dbc-8798-ef7ae9152518\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " Oct 30 18:38:57.545742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545514 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-data" pod "30b7c407-135e-45ee-96b6-b09469351896" (UID: "30b7c407-135e-45ee-96b6-b09469351896"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.545742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545520 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-socket\") pod \"30b7c407-135e-45ee-96b6-b09469351896\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " Oct 30 18:38:57.545742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545556 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-socket\") pod \"60201992-7051-4dbc-8798-ef7ae9152518\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " Oct 30 18:38:57.545742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545593 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-data\") pod \"0086e9f0-1201-4263-b710-5417225349ee\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " Oct 30 18:38:57.545742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545625 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-certs\") pod \"30b7c407-135e-45ee-96b6-b09469351896\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " Oct 30 18:38:57.545742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545667 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-istio-token\" (UniqueName: \"kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-istio-token\") pod \"60201992-7051-4dbc-8798-ef7ae9152518\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " Oct 30 18:38:57.545742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545700 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0086e9f0-1201-4263-b710-5417225349ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0086e9f0-1201-4263-b710-5417225349ee-istiod-ca-cert\") pod \"0086e9f0-1201-4263-b710-5417225349ee\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " Oct 30 18:38:57.545742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545732 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-envoy\") pod \"30b7c407-135e-45ee-96b6-b09469351896\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " Oct 30 18:38:57.546411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545758 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60201992-7051-4dbc-8798-ef7ae9152518-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60201992-7051-4dbc-8798-ef7ae9152518-istiod-ca-cert\") pod \"60201992-7051-4dbc-8798-ef7ae9152518\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " Oct 30 18:38:57.546411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545809 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-istio-token\" (UniqueName: \"kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-istio-token\") pod \"0086e9f0-1201-4263-b710-5417225349ee\" (UID: \"0086e9f0-1201-4263-b710-5417225349ee\") " Oct 30 18:38:57.546411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.545877 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-kube-api-access-qrb6k\" (UniqueName: \"kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-kube-api-access-qrb6k\") pod \"30b7c407-135e-45ee-96b6-b09469351896\" (UID: \"30b7c407-135e-45ee-96b6-b09469351896\") " Oct 30 18:38:57.546411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546007 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-certs\") pod \"60201992-7051-4dbc-8798-ef7ae9152518\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " Oct 30 18:38:57.546411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546080 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-kube-api-access-7ts9p\" (UniqueName: \"kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-kube-api-access-7ts9p\") pod \"60201992-7051-4dbc-8798-ef7ae9152518\" (UID: \"60201992-7051-4dbc-8798-ef7ae9152518\") " Oct 30 18:38:57.546411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546188 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-socket" pod "30b7c407-135e-45ee-96b6-b09469351896" (UID: "30b7c407-135e-45ee-96b6-b09469351896"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.546411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546278 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-certs" pod "30b7c407-135e-45ee-96b6-b09469351896" (UID: "30b7c407-135e-45ee-96b6-b09469351896"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.546411 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546298 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-socket" pod "60201992-7051-4dbc-8798-ef7ae9152518" (UID: "60201992-7051-4dbc-8798-ef7ae9152518"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.546821 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546588 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-data" pod "60201992-7051-4dbc-8798-ef7ae9152518" (UID: "60201992-7051-4dbc-8798-ef7ae9152518"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.546881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546844 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.546881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546864 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.547015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546878 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.547015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546893 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.547015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546910 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.547015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546923 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.547015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546955 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b7c407-135e-45ee-96b6-b09469351896-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.547015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546970 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.547015 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.546984 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.547454 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.547436 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-data" pod "0086e9f0-1201-4263-b710-5417225349ee" (UID: "0086e9f0-1201-4263-b710-5417225349ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.548121 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.547766 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-credential-socket" pod "30b7c407-135e-45ee-96b6-b09469351896" (UID: "30b7c407-135e-45ee-96b6-b09469351896"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.548121 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.547914 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-certs" pod "60201992-7051-4dbc-8798-ef7ae9152518" (UID: "60201992-7051-4dbc-8798-ef7ae9152518"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.548121 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.548111 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-credential-socket" pod "60201992-7051-4dbc-8798-ef7ae9152518" (UID: "60201992-7051-4dbc-8798-ef7ae9152518"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.548661 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.548634 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0086e9f0-1201-4263-b710-5417225349ee-istiod-ca-cert" pod "0086e9f0-1201-4263-b710-5417225349ee" (UID: "0086e9f0-1201-4263-b710-5417225349ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:57.548961 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.548923 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60201992-7051-4dbc-8798-ef7ae9152518-istiod-ca-cert" pod "60201992-7051-4dbc-8798-ef7ae9152518" (UID: "60201992-7051-4dbc-8798-ef7ae9152518"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:57.550558 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.550531 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30b7c407-135e-45ee-96b6-b09469351896-istio-podinfo" pod "30b7c407-135e-45ee-96b6-b09469351896" (UID: "30b7c407-135e-45ee-96b6-b09469351896"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:57.550895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.550834 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-istio-token" pod "0086e9f0-1201-4263-b710-5417225349ee" (UID: "0086e9f0-1201-4263-b710-5417225349ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.551344 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.551311 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-envoy" pod "30b7c407-135e-45ee-96b6-b09469351896" (UID: "30b7c407-135e-45ee-96b6-b09469351896"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.551422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.551372 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-kube-api-access-qrb6k" pod "30b7c407-135e-45ee-96b6-b09469351896" (UID: "30b7c407-135e-45ee-96b6-b09469351896"). InnerVolumeSpecName "kube-api-access-qrb6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.551536 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.551509 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-kube-api-access-7ts9p" pod "60201992-7051-4dbc-8798-ef7ae9152518" (UID: "60201992-7051-4dbc-8798-ef7ae9152518"). InnerVolumeSpecName "kube-api-access-7ts9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.552342 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.552293 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60201992-7051-4dbc-8798-ef7ae9152518-istio-podinfo" pod "60201992-7051-4dbc-8798-ef7ae9152518" (UID: "60201992-7051-4dbc-8798-ef7ae9152518"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:57.552342 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.552294 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-envoy" pod "60201992-7051-4dbc-8798-ef7ae9152518" (UID: "60201992-7051-4dbc-8798-ef7ae9152518"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.552531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.552394 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0086e9f0-1201-4263-b710-5417225349ee-istio-podinfo" pod "0086e9f0-1201-4263-b710-5417225349ee" (UID: "0086e9f0-1201-4263-b710-5417225349ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:57.552692 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.552671 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-kube-api-access-vpdhs" pod "0086e9f0-1201-4263-b710-5417225349ee" (UID: "0086e9f0-1201-4263-b710-5417225349ee"). InnerVolumeSpecName "kube-api-access-vpdhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.552998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.552921 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-istio-token" pod "30b7c407-135e-45ee-96b6-b09469351896" (UID: "30b7c407-135e-45ee-96b6-b09469351896"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.554039 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.554013 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-istio-token" pod "60201992-7051-4dbc-8798-ef7ae9152518" (UID: "60201992-7051-4dbc-8798-ef7ae9152518"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:57.554227 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.554201 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-envoy" pod "0086e9f0-1201-4263-b710-5417225349ee" (UID: "0086e9f0-1201-4263-b710-5417225349ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:57.569602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.568795 4883 scope.go:122] "RemoveContainer" containerID="9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568" Oct 30 18:38:57.569602 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:38:57.569319 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568\": container with ID starting with 9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568 not found: ID does not exist" containerID="9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568" Oct 30 18:38:57.569602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.569469 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568"} err="failed to get container status \"9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568\": rpc error: code = NotFound desc = could not find container \"9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568\": container with ID starting with 9b67bf69e40d22baf9d8ff8a9edb23af9bea1e92a1dc3903bf385e872dfbd568 not found: ID does not exist" Oct 30 18:38:57.569602 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.569534 4883 scope.go:122] "RemoveContainer" containerID="8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218" Oct 30 18:38:57.588655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.588630 4883 scope.go:122] "RemoveContainer" containerID="8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218" Oct 30 18:38:57.589093 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:38:57.589070 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218\": container with ID starting with 8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218 not found: ID does not exist" containerID="8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218" Oct 30 18:38:57.589203 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.589108 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218"} err="failed to get container status \"8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218\": rpc error: code = NotFound desc = could not find container \"8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218\": container with ID starting with 8a420e9b18e508a3350151e9171cfb27015c83d2d625a051cdf8a59ba28a0218 not found: ID does not exist" Oct 30 18:38:57.648578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648506 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648535 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648547 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648555 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b7c407-135e-45ee-96b6-b09469351896-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648564 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0086e9f0-1201-4263-b710-5417225349ee-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648572 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648578 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648579 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0086e9f0-1201-4263-b710-5417225349ee-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648587 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b7c407-135e-45ee-96b6-b09469351896-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648595 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60201992-7051-4dbc-8798-ef7ae9152518-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648603 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648613 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrb6k\" (UniqueName: \"kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-kube-api-access-qrb6k\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648623 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648633 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ts9p\" (UniqueName: \"kubernetes.io/projected/60201992-7051-4dbc-8798-ef7ae9152518-kube-api-access-7ts9p\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648641 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0086e9f0-1201-4263-b710-5417225349ee-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648648 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60201992-7051-4dbc-8798-ef7ae9152518-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648656 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60201992-7051-4dbc-8798-ef7ae9152518-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648665 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpdhs\" (UniqueName: \"kubernetes.io/projected/0086e9f0-1201-4263-b710-5417225349ee-kube-api-access-vpdhs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.648930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.648674 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b7c407-135e-45ee-96b6-b09469351896-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:38:57.890594 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.890559 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj"] Oct 30 18:38:57.905172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.905082 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-z5gbj"] Oct 30 18:38:57.947232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.947192 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r"] Oct 30 18:38:57.950246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.950217 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-57h5r"] Oct 30 18:38:57.988675 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:57.988640 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8"] Oct 30 18:38:58.011188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:58.011159 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6e8fca-openshift-default-77759666dc-72gb8"] Oct 30 18:38:59.177577 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:59.177540 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0086e9f0-1201-4263-b710-5417225349ee" path="/var/lib/kubelet/pods/0086e9f0-1201-4263-b710-5417225349ee/volumes" Oct 30 18:38:59.178135 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:59.178117 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b7c407-135e-45ee-96b6-b09469351896" path="/var/lib/kubelet/pods/30b7c407-135e-45ee-96b6-b09469351896/volumes" Oct 30 18:38:59.178613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:38:59.178597 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60201992-7051-4dbc-8798-ef7ae9152518" path="/var/lib/kubelet/pods/60201992-7051-4dbc-8798-ef7ae9152518/volumes" Oct 30 18:39:05.078614 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:39:05.078569 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:39:07.480652 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:07.480610 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5"] Oct 30 18:39:07.491031 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:07.490986 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-58d08b-m8nn5"] Oct 30 18:39:09.178382 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:09.178346 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71abc140-6296-4993-b427-61a9977cabde" path="/var/lib/kubelet/pods/71abc140-6296-4993-b427-61a9977cabde/volumes" Oct 30 18:39:11.429130 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:11.429097 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck"] Oct 30 18:39:11.429608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:11.429329 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" containerID="cri-o://20a2f17b46487cc0b08c1199b789b708a6f3cb4e6e40323494f9f6e145984fd7" gracePeriod=30 Oct 30 18:39:12.131701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.131660 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46"] Oct 30 18:39:12.132261 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132242 4883 cpu_manager.go:414] "containerMap: removing container" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" Oct 30 18:39:12.132386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132262 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" Oct 30 18:39:12.132386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132280 4883 cpu_manager.go:414] "containerMap: removing container" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" Oct 30 18:39:12.132386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132287 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" Oct 30 18:39:12.132386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132312 4883 cpu_manager.go:414] "containerMap: removing container" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" Oct 30 18:39:12.132386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132320 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" Oct 30 18:39:12.132386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132331 4883 cpu_manager.go:414] "containerMap: removing container" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" Oct 30 18:39:12.132386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132338 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" Oct 30 18:39:12.132386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132351 4883 cpu_manager.go:414] "containerMap: removing container" podUID="71abc140-6296-4993-b427-61a9977cabde" containerName="revoke-keys" Oct 30 18:39:12.132386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132358 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71abc140-6296-4993-b427-61a9977cabde" containerName="revoke-keys" Oct 30 18:39:12.132386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132377 4883 cpu_manager.go:414] "containerMap: removing container" podUID="0086e9f0-1201-4263-b710-5417225349ee" containerName="istio-proxy" Oct 30 18:39:12.132386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132384 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0086e9f0-1201-4263-b710-5417225349ee" containerName="istio-proxy" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132395 4883 cpu_manager.go:414] "containerMap: removing container" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerName="main" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132402 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerName="main" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132451 4883 cpu_manager.go:414] "containerMap: removing container" podUID="85b87546-1283-431a-b62a-2718b90bc3cb" containerName="main" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132460 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b87546-1283-431a-b62a-2718b90bc3cb" containerName="main" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132475 4883 cpu_manager.go:414] "containerMap: removing container" podUID="a52b900a-f62c-45e5-bc7b-b3bb1afb7075" containerName="revoke-keys" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132484 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52b900a-f62c-45e5-bc7b-b3bb1afb7075" containerName="revoke-keys" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132498 4883 cpu_manager.go:414] "containerMap: removing container" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132506 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132637 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b87546-1283-431a-b62a-2718b90bc3cb" containerName="main" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132652 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52b900a-f62c-45e5-bc7b-b3bb1afb7075" containerName="revoke-keys" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132663 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b7c407-135e-45ee-96b6-b09469351896" containerName="istio-proxy" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132677 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="60201992-7051-4dbc-8798-ef7ae9152518" containerName="istio-proxy" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132686 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="f847895f-c06c-4e79-8f10-9305e980856a" containerName="istio-proxy" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132698 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5b4960-719d-4f28-893f-a6d4a7e41369" containerName="istio-proxy" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132709 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b1ce30-99a1-4d3b-8bb1-f41c9705a9b9" containerName="istio-proxy" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132720 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="71abc140-6296-4993-b427-61a9977cabde" containerName="revoke-keys" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132730 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="0086e9f0-1201-4263-b710-5417225349ee" containerName="istio-proxy" Oct 30 18:39:12.132984 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.132739 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="df3fde75-25a2-4fde-b7a4-78e412c6c696" containerName="main" Oct 30 18:39:12.138246 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.138224 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:12.176968 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.170242 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:39:12.184089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.183820 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:39:12.184089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.183900 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:39:12.184089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.184052 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:39:12.219396 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.219348 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46"] Oct 30 18:39:12.235277 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.235228 4883 patch_prober.go:29] interesting pod/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:39:12.235618 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.235584 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:39:12.293840 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.293800 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46\" (UID: \"1ddcfd76-2685-4c89-af66-2ba84b75faae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:12.294085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.293856 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8m2x\" (UniqueName: \"kubernetes.io/projected/1ddcfd76-2685-4c89-af66-2ba84b75faae-kube-api-access-c8m2x\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46\" (UID: \"1ddcfd76-2685-4c89-af66-2ba84b75faae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:12.294085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.294048 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46\" (UID: \"1ddcfd76-2685-4c89-af66-2ba84b75faae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:12.395209 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.395116 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46\" (UID: \"1ddcfd76-2685-4c89-af66-2ba84b75faae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:12.395369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.395283 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46\" (UID: \"1ddcfd76-2685-4c89-af66-2ba84b75faae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:12.395369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.395307 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8m2x\" (UniqueName: \"kubernetes.io/projected/1ddcfd76-2685-4c89-af66-2ba84b75faae-kube-api-access-c8m2x\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46\" (UID: \"1ddcfd76-2685-4c89-af66-2ba84b75faae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:12.396004 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.395982 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46\" (UID: \"1ddcfd76-2685-4c89-af66-2ba84b75faae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:12.397595 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.397565 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46\" (UID: \"1ddcfd76-2685-4c89-af66-2ba84b75faae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:12.424520 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.424487 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8m2x\" (UniqueName: \"kubernetes.io/projected/1ddcfd76-2685-4c89-af66-2ba84b75faae-kube-api-access-c8m2x\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46\" (UID: \"1ddcfd76-2685-4c89-af66-2ba84b75faae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:12.453203 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.453169 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:12.765264 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:12.765238 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46"] Oct 30 18:39:12.767922 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:39:12.767886 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ddcfd76_2685_4c89_af66_2ba84b75faae.slice/crio-ed6fb2f36292559f518ecad5a1a7c20d32fa7847fcdb270bb468083246062e47 WatchSource:0}: Error finding container ed6fb2f36292559f518ecad5a1a7c20d32fa7847fcdb270bb468083246062e47: Status 404 returned error can't find the container with id ed6fb2f36292559f518ecad5a1a7c20d32fa7847fcdb270bb468083246062e47 Oct 30 18:39:13.666572 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:13.666486 4883 generic.go:356] "Generic (PLEG): container finished" podID="1ddcfd76-2685-4c89-af66-2ba84b75faae" containerID="6731998cfd2b553c2fe546d039f8bc827a7ce1ae0108205e27d729292abdbdb0" exitCode=0 Oct 30 18:39:13.666977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:13.666569 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" event={"ID":"1ddcfd76-2685-4c89-af66-2ba84b75faae","Type":"ContainerDied","Data":"6731998cfd2b553c2fe546d039f8bc827a7ce1ae0108205e27d729292abdbdb0"} Oct 30 18:39:13.666977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:13.666621 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" event={"ID":"1ddcfd76-2685-4c89-af66-2ba84b75faae","Type":"ContainerStarted","Data":"ed6fb2f36292559f518ecad5a1a7c20d32fa7847fcdb270bb468083246062e47"} Oct 30 18:39:14.833983 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:14.833947 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:14.927316 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:14.927222 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddcfd76-2685-4c89-af66-2ba84b75faae-kube-api-access-c8m2x\" (UniqueName: \"kubernetes.io/projected/1ddcfd76-2685-4c89-af66-2ba84b75faae-kube-api-access-c8m2x\") pod \"1ddcfd76-2685-4c89-af66-2ba84b75faae\" (UID: \"1ddcfd76-2685-4c89-af66-2ba84b75faae\") " Oct 30 18:39:14.927495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:14.927403 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-xks-ca\") pod \"1ddcfd76-2685-4c89-af66-2ba84b75faae\" (UID: \"1ddcfd76-2685-4c89-af66-2ba84b75faae\") " Oct 30 18:39:14.927495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:14.927483 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-openshift-ca\") pod \"1ddcfd76-2685-4c89-af66-2ba84b75faae\" (UID: \"1ddcfd76-2685-4c89-af66-2ba84b75faae\") " Oct 30 18:39:14.927926 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:14.927895 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-openshift-ca" pod "1ddcfd76-2685-4c89-af66-2ba84b75faae" (UID: "1ddcfd76-2685-4c89-af66-2ba84b75faae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:14.929879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:14.929857 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-xks-ca" pod "1ddcfd76-2685-4c89-af66-2ba84b75faae" (UID: "1ddcfd76-2685-4c89-af66-2ba84b75faae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:14.929978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:14.929914 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddcfd76-2685-4c89-af66-2ba84b75faae-kube-api-access-c8m2x" pod "1ddcfd76-2685-4c89-af66-2ba84b75faae" (UID: "1ddcfd76-2685-4c89-af66-2ba84b75faae"). InnerVolumeSpecName "kube-api-access-c8m2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:15.028837 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:15.028804 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:15.028837 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:15.028834 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8m2x\" (UniqueName: \"kubernetes.io/projected/1ddcfd76-2685-4c89-af66-2ba84b75faae-kube-api-access-c8m2x\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:15.028837 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:15.028844 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ddcfd76-2685-4c89-af66-2ba84b75faae-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:15.681254 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:15.681226 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" Oct 30 18:39:15.681437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:15.681266 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46" event={"ID":"1ddcfd76-2685-4c89-af66-2ba84b75faae","Type":"ContainerDied","Data":"ed6fb2f36292559f518ecad5a1a7c20d32fa7847fcdb270bb468083246062e47"} Oct 30 18:39:15.681437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:15.681321 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed6fb2f36292559f518ecad5a1a7c20d32fa7847fcdb270bb468083246062e47" Oct 30 18:39:16.689226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.689198 4883 generic.go:356] "Generic (PLEG): container finished" podID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerID="20a2f17b46487cc0b08c1199b789b708a6f3cb4e6e40323494f9f6e145984fd7" exitCode=0 Oct 30 18:39:16.689604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.689278 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" event={"ID":"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470","Type":"ContainerDied","Data":"20a2f17b46487cc0b08c1199b789b708a6f3cb4e6e40323494f9f6e145984fd7"} Oct 30 18:39:16.689604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.689336 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" event={"ID":"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470","Type":"ContainerDied","Data":"e50a02a9e4a34ebbb25bf5decf255d3d3f64b569423dab4661a63c06892874eb"} Oct 30 18:39:16.689604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.689367 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e50a02a9e4a34ebbb25bf5decf255d3d3f64b569423dab4661a63c06892874eb" Oct 30 18:39:16.689604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.689381 4883 scope.go:122] "RemoveContainer" containerID="20da60c9571f2a60babdc1170cc039a0f239d2b0f5e116c2bb210a06900a642d" Oct 30 18:39:16.701888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.701867 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:39:16.849780 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.849748 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-kube-api-access-hxqqz\" (UniqueName: \"kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-kube-api-access-hxqqz\") pod \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " Oct 30 18:39:16.850001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.849861 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-credential-socket\") pod \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " Oct 30 18:39:16.850001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.849892 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istiod-ca-cert\") pod \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " Oct 30 18:39:16.850001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.849925 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-token\" (UniqueName: \"kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-token\") pod \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " Oct 30 18:39:16.850001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.849995 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-data\") pod \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " Oct 30 18:39:16.850231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850078 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-certs\") pod \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " Oct 30 18:39:16.850231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850126 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-podinfo\") pod \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " Oct 30 18:39:16.850231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850162 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-socket\") pod \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " Oct 30 18:39:16.850392 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850232 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-envoy\") pod \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\" (UID: \"cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470\") " Oct 30 18:39:16.850392 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850236 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-credential-socket" pod "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" (UID: "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:16.850392 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850320 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-data" pod "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" (UID: "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:16.850392 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850356 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-certs" pod "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" (UID: "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:16.850613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850392 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istiod-ca-cert" pod "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" (UID: "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:16.850613 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850463 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-socket" pod "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" (UID: "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:16.850846 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850827 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:16.850906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850852 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:16.850906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850866 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:16.850906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850878 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:16.850906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.850891 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:16.852174 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.852153 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-token" pod "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" (UID: "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:16.852290 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.852270 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-podinfo" pod "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" (UID: "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:39:16.852581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.852558 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-kube-api-access-hxqqz" pod "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" (UID: "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470"). InnerVolumeSpecName "kube-api-access-hxqqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:16.852691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.852623 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-envoy" pod "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" (UID: "cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:16.952318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.952237 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:16.952318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.952267 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxqqz\" (UniqueName: \"kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-kube-api-access-hxqqz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:16.952318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.952278 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:16.952318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:16.952287 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:17.698109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:17.698077 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck" Oct 30 18:39:17.803641 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:17.803606 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck"] Oct 30 18:39:17.806262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:17.806238 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-a1a31f-openshift-default-69dd547b5-rzdck"] Oct 30 18:39:19.180148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:19.178415 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" path="/var/lib/kubelet/pods/cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470/volumes" Oct 30 18:39:27.462188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.462138 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7"] Oct 30 18:39:27.462673 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.462658 4883 cpu_manager.go:414] "containerMap: removing container" podUID="1ddcfd76-2685-4c89-af66-2ba84b75faae" containerName="revoke-keys" Oct 30 18:39:27.462722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.462676 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddcfd76-2685-4c89-af66-2ba84b75faae" containerName="revoke-keys" Oct 30 18:39:27.462722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.462701 4883 cpu_manager.go:414] "containerMap: removing container" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" Oct 30 18:39:27.462722 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.462706 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" Oct 30 18:39:27.462826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.462730 4883 cpu_manager.go:414] "containerMap: removing container" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" Oct 30 18:39:27.462826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.462753 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" Oct 30 18:39:27.462826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.462823 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" Oct 30 18:39:27.462930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.462832 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddcfd76-2685-4c89-af66-2ba84b75faae" containerName="revoke-keys" Oct 30 18:39:27.462930 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.462841 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2ea73e-ecbd-4e79-8e24-d6e2a5ef3470" containerName="istio-proxy" Oct 30 18:39:27.465924 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.465900 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:27.483320 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.483290 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:39:27.497119 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.497082 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:39:27.500719 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.500678 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7"] Oct 30 18:39:27.504479 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.504447 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:39:27.504685 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.504535 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:39:27.539861 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.539811 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrdl2\" (UniqueName: \"kubernetes.io/projected/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-kube-api-access-lrdl2\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7\" (UID: \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:27.540092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.539979 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7\" (UID: \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:27.540092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.540053 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7\" (UID: \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:27.641385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.641342 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrdl2\" (UniqueName: \"kubernetes.io/projected/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-kube-api-access-lrdl2\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7\" (UID: \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:27.641581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.641457 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7\" (UID: \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:27.641581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.641485 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7\" (UID: \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:27.642166 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.642134 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7\" (UID: \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:27.643895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.643869 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7\" (UID: \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:27.667058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.667019 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrdl2\" (UniqueName: \"kubernetes.io/projected/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-kube-api-access-lrdl2\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7\" (UID: \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:27.781702 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:27.781616 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:28.097576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:28.097540 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7"] Oct 30 18:39:28.099094 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:39:28.099057 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0af39f_af6e_4fa7_9fcc_2ea94222a288.slice/crio-8cdefe5588de02c00e4f08a63a76d27c9ac03b1306083655f7dc1373ce8bd1a8 WatchSource:0}: Error finding container 8cdefe5588de02c00e4f08a63a76d27c9ac03b1306083655f7dc1373ce8bd1a8: Status 404 returned error can't find the container with id 8cdefe5588de02c00e4f08a63a76d27c9ac03b1306083655f7dc1373ce8bd1a8 Oct 30 18:39:28.791835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:28.791743 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" event={"ID":"dc0af39f-af6e-4fa7-9fcc-2ea94222a288","Type":"ContainerStarted","Data":"b77657058c45ef1b11e0c1edaf186e45f5d0aa706ca731414bb378d354b27916"} Oct 30 18:39:28.792437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:28.792403 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" event={"ID":"dc0af39f-af6e-4fa7-9fcc-2ea94222a288","Type":"ContainerStarted","Data":"8cdefe5588de02c00e4f08a63a76d27c9ac03b1306083655f7dc1373ce8bd1a8"} Oct 30 18:39:28.841774 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:28.841728 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" podStartSLOduration=1.841714295 podStartE2EDuration="1.841714295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:28.841056202 +0000 UTC m=+2827.798060038" watchObservedRunningTime="2026-10-30 18:39:28.841714295 +0000 UTC m=+2827.798718130" Oct 30 18:39:29.800066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:29.800032 4883 generic.go:356] "Generic (PLEG): container finished" podID="dc0af39f-af6e-4fa7-9fcc-2ea94222a288" containerID="b77657058c45ef1b11e0c1edaf186e45f5d0aa706ca731414bb378d354b27916" exitCode=0 Oct 30 18:39:29.800483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:29.800118 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" event={"ID":"dc0af39f-af6e-4fa7-9fcc-2ea94222a288","Type":"ContainerDied","Data":"b77657058c45ef1b11e0c1edaf186e45f5d0aa706ca731414bb378d354b27916"} Oct 30 18:39:30.967270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:30.967240 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:31.100000 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:31.099958 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-openshift-ca\") pod \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\" (UID: \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\") " Oct 30 18:39:31.100182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:31.100029 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-kube-api-access-lrdl2\" (UniqueName: \"kubernetes.io/projected/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-kube-api-access-lrdl2\") pod \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\" (UID: \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\") " Oct 30 18:39:31.100182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:31.100109 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-xks-ca\") pod \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\" (UID: \"dc0af39f-af6e-4fa7-9fcc-2ea94222a288\") " Oct 30 18:39:31.100368 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:31.100345 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-openshift-ca" pod "dc0af39f-af6e-4fa7-9fcc-2ea94222a288" (UID: "dc0af39f-af6e-4fa7-9fcc-2ea94222a288"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:31.100615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:31.100600 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:31.102266 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:31.102243 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-kube-api-access-lrdl2" pod "dc0af39f-af6e-4fa7-9fcc-2ea94222a288" (UID: "dc0af39f-af6e-4fa7-9fcc-2ea94222a288"). InnerVolumeSpecName "kube-api-access-lrdl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:31.102383 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:31.102329 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-xks-ca" pod "dc0af39f-af6e-4fa7-9fcc-2ea94222a288" (UID: "dc0af39f-af6e-4fa7-9fcc-2ea94222a288"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:31.201772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:31.201726 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrdl2\" (UniqueName: \"kubernetes.io/projected/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-kube-api-access-lrdl2\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:31.201772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:31.201762 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc0af39f-af6e-4fa7-9fcc-2ea94222a288-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:31.815214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:31.815179 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" Oct 30 18:39:31.815214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:31.815206 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7" event={"ID":"dc0af39f-af6e-4fa7-9fcc-2ea94222a288","Type":"ContainerDied","Data":"8cdefe5588de02c00e4f08a63a76d27c9ac03b1306083655f7dc1373ce8bd1a8"} Oct 30 18:39:31.815459 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:31.815253 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cdefe5588de02c00e4f08a63a76d27c9ac03b1306083655f7dc1373ce8bd1a8" Oct 30 18:39:35.078521 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:39:35.078483 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:39:38.225560 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:38.225526 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46"] Oct 30 18:39:38.238442 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:38.238410 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e77b1d-6xx46"] Oct 30 18:39:39.177816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:39.177782 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ddcfd76-2685-4c89-af66-2ba84b75faae" path="/var/lib/kubelet/pods/1ddcfd76-2685-4c89-af66-2ba84b75faae/volumes" Oct 30 18:39:39.776625 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:39.776572 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7"] Oct 30 18:39:39.783714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:39.783675 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-50e9fc-wt5c7"] Oct 30 18:39:41.178576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:41.178541 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc0af39f-af6e-4fa7-9fcc-2ea94222a288" path="/var/lib/kubelet/pods/dc0af39f-af6e-4fa7-9fcc-2ea94222a288/volumes" Oct 30 18:39:46.301330 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:46.301292 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9"] Oct 30 18:39:46.301841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:46.301518 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" podUID="3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" containerName="istio-proxy" containerID="cri-o://dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7" gracePeriod=30 Oct 30 18:39:46.930107 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:46.930073 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn"] Oct 30 18:39:46.930615 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:46.930601 4883 cpu_manager.go:414] "containerMap: removing container" podUID="dc0af39f-af6e-4fa7-9fcc-2ea94222a288" containerName="revoke-keys" Oct 30 18:39:46.930666 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:46.930617 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0af39f-af6e-4fa7-9fcc-2ea94222a288" containerName="revoke-keys" Oct 30 18:39:46.930723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:46.930714 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc0af39f-af6e-4fa7-9fcc-2ea94222a288" containerName="revoke-keys" Oct 30 18:39:46.935208 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:46.935188 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:46.954884 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:46.954861 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:39:46.955716 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:46.955699 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:39:46.955851 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:46.955831 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:39:46.968669 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:46.968645 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:39:46.973564 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:46.973534 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn"] Oct 30 18:39:47.020449 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:47.020419 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97h5\" (UniqueName: \"kubernetes.io/projected/861fdfb3-c9f1-4301-b316-3f5ddfa98496-kube-api-access-b97h5\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn\" (UID: \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:47.020658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:47.020638 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn\" (UID: \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:47.020712 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:47.020698 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn\" (UID: \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:47.122176 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:47.122137 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn\" (UID: \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:47.122347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:47.122199 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn\" (UID: \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:47.122347 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:47.122263 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97h5\" (UniqueName: \"kubernetes.io/projected/861fdfb3-c9f1-4301-b316-3f5ddfa98496-kube-api-access-b97h5\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn\" (UID: \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:47.122877 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:47.122852 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn\" (UID: \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:47.124695 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:47.124676 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn\" (UID: \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:47.158390 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:47.158359 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97h5\" (UniqueName: \"kubernetes.io/projected/861fdfb3-c9f1-4301-b316-3f5ddfa98496-kube-api-access-b97h5\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn\" (UID: \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:47.248198 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:47.248105 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:47.576087 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:39:47.576039 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861fdfb3_c9f1_4301_b316_3f5ddfa98496.slice/crio-5aead47bed6fbc0217463ef88a850d8a14c6bc21bbb28011efce23bfc1fe4fd5 WatchSource:0}: Error finding container 5aead47bed6fbc0217463ef88a850d8a14c6bc21bbb28011efce23bfc1fe4fd5: Status 404 returned error can't find the container with id 5aead47bed6fbc0217463ef88a850d8a14c6bc21bbb28011efce23bfc1fe4fd5 Oct 30 18:39:47.579732 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:47.579704 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn"] Oct 30 18:39:48.045810 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:48.045772 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" event={"ID":"861fdfb3-c9f1-4301-b316-3f5ddfa98496","Type":"ContainerStarted","Data":"fd2a9a7d7bf50c2477df1cc6aa50180e751c0d62eeff1e79fe42ba84e8ef7dc8"} Oct 30 18:39:48.046005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:48.045825 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" event={"ID":"861fdfb3-c9f1-4301-b316-3f5ddfa98496","Type":"ContainerStarted","Data":"5aead47bed6fbc0217463ef88a850d8a14c6bc21bbb28011efce23bfc1fe4fd5"} Oct 30 18:39:48.108733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:48.108680 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" podStartSLOduration=2.10866513 podStartE2EDuration="2.10866513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:48.105804319 +0000 UTC m=+2847.062808157" watchObservedRunningTime="2026-10-30 18:39:48.10866513 +0000 UTC m=+2847.065668994" Oct 30 18:39:49.053819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:49.053782 4883 generic.go:356] "Generic (PLEG): container finished" podID="861fdfb3-c9f1-4301-b316-3f5ddfa98496" containerID="fd2a9a7d7bf50c2477df1cc6aa50180e751c0d62eeff1e79fe42ba84e8ef7dc8" exitCode=0 Oct 30 18:39:49.054280 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:49.053883 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" event={"ID":"861fdfb3-c9f1-4301-b316-3f5ddfa98496","Type":"ContainerDied","Data":"fd2a9a7d7bf50c2477df1cc6aa50180e751c0d62eeff1e79fe42ba84e8ef7dc8"} Oct 30 18:39:49.950640 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:49.950606 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls"] Oct 30 18:39:49.954423 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:49.954405 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:49.980794 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:49.980747 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls"] Oct 30 18:39:50.065133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.065100 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnv7s\" (UniqueName: \"kubernetes.io/projected/4734f1ff-9784-404b-ab17-94075cdae588-kube-api-access-bnv7s\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls\" (UID: \"4734f1ff-9784-404b-ab17-94075cdae588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:50.065533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.065242 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls\" (UID: \"4734f1ff-9784-404b-ab17-94075cdae588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:50.065533 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.065353 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls\" (UID: \"4734f1ff-9784-404b-ab17-94075cdae588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:50.167650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.166864 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls\" (UID: \"4734f1ff-9784-404b-ab17-94075cdae588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:50.167650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.167027 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnv7s\" (UniqueName: \"kubernetes.io/projected/4734f1ff-9784-404b-ab17-94075cdae588-kube-api-access-bnv7s\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls\" (UID: \"4734f1ff-9784-404b-ab17-94075cdae588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:50.167650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.167102 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls\" (UID: \"4734f1ff-9784-404b-ab17-94075cdae588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:50.168719 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.168689 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls\" (UID: \"4734f1ff-9784-404b-ab17-94075cdae588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:50.171360 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.171197 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls\" (UID: \"4734f1ff-9784-404b-ab17-94075cdae588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:50.198001 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.197964 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnv7s\" (UniqueName: \"kubernetes.io/projected/4734f1ff-9784-404b-ab17-94075cdae588-kube-api-access-bnv7s\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls\" (UID: \"4734f1ff-9784-404b-ab17-94075cdae588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:50.219338 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.219277 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:50.268271 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.268233 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:50.370953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.370163 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/861fdfb3-c9f1-4301-b316-3f5ddfa98496-kube-api-access-b97h5\" (UniqueName: \"kubernetes.io/projected/861fdfb3-c9f1-4301-b316-3f5ddfa98496-kube-api-access-b97h5\") pod \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\" (UID: \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\") " Oct 30 18:39:50.370953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.370287 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-xks-ca\") pod \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\" (UID: \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\") " Oct 30 18:39:50.370953 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.370364 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-openshift-ca\") pod \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\" (UID: \"861fdfb3-c9f1-4301-b316-3f5ddfa98496\") " Oct 30 18:39:50.371315 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.371288 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-openshift-ca" pod "861fdfb3-c9f1-4301-b316-3f5ddfa98496" (UID: "861fdfb3-c9f1-4301-b316-3f5ddfa98496"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:50.373127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.373102 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-xks-ca" pod "861fdfb3-c9f1-4301-b316-3f5ddfa98496" (UID: "861fdfb3-c9f1-4301-b316-3f5ddfa98496"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:50.373256 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.373242 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861fdfb3-c9f1-4301-b316-3f5ddfa98496-kube-api-access-b97h5" pod "861fdfb3-c9f1-4301-b316-3f5ddfa98496" (UID: "861fdfb3-c9f1-4301-b316-3f5ddfa98496"). InnerVolumeSpecName "kube-api-access-b97h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:50.471954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.471834 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:50.471954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.471879 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b97h5\" (UniqueName: \"kubernetes.io/projected/861fdfb3-c9f1-4301-b316-3f5ddfa98496-kube-api-access-b97h5\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:50.471954 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.471896 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/861fdfb3-c9f1-4301-b316-3f5ddfa98496-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:50.588829 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:50.588799 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls"] Oct 30 18:39:50.589693 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:39:50.589667 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4734f1ff_9784_404b_ab17_94075cdae588.slice/crio-7b6ee89650000dc24e2b80ef661726e48a656761ab06ec7160c2916b78576744 WatchSource:0}: Error finding container 7b6ee89650000dc24e2b80ef661726e48a656761ab06ec7160c2916b78576744: Status 404 returned error can't find the container with id 7b6ee89650000dc24e2b80ef661726e48a656761ab06ec7160c2916b78576744 Oct 30 18:39:51.067656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.067561 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" Oct 30 18:39:51.067656 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.067588 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn" event={"ID":"861fdfb3-c9f1-4301-b316-3f5ddfa98496","Type":"ContainerDied","Data":"5aead47bed6fbc0217463ef88a850d8a14c6bc21bbb28011efce23bfc1fe4fd5"} Oct 30 18:39:51.068235 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.067669 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aead47bed6fbc0217463ef88a850d8a14c6bc21bbb28011efce23bfc1fe4fd5" Oct 30 18:39:51.069352 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.069324 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" event={"ID":"4734f1ff-9784-404b-ab17-94075cdae588","Type":"ContainerStarted","Data":"f29cb08f4912f898b4beeee3f022830ea00682ea88685b3714cf452911aa0e97"} Oct 30 18:39:51.069492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.069372 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" event={"ID":"4734f1ff-9784-404b-ab17-94075cdae588","Type":"ContainerStarted","Data":"7b6ee89650000dc24e2b80ef661726e48a656761ab06ec7160c2916b78576744"} Oct 30 18:39:51.129744 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.129687 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" podStartSLOduration=2.129671062 podStartE2EDuration="2.129671062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:51.126536054 +0000 UTC m=+2850.083539890" watchObservedRunningTime="2026-10-30 18:39:51.129671062 +0000 UTC m=+2850.086674898" Oct 30 18:39:51.570443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.570412 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:39:51.689280 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.689242 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istiod-ca-cert\") pod \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " Oct 30 18:39:51.689475 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.689326 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-socket\") pod \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " Oct 30 18:39:51.689475 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.689391 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-kube-api-access-4qrs6\" (UniqueName: \"kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-kube-api-access-4qrs6\") pod \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " Oct 30 18:39:51.689475 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.689441 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-podinfo\") pod \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " Oct 30 18:39:51.689667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.689494 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-token\" (UniqueName: \"kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-token\") pod \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " Oct 30 18:39:51.689667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.689534 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-credential-socket\") pod \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " Oct 30 18:39:51.689667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.689560 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-data\") pod \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " Oct 30 18:39:51.689667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.689590 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-envoy\") pod \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " Oct 30 18:39:51.689888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.689663 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-socket" pod "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" (UID: "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.689888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.689709 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istiod-ca-cert" pod "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" (UID: "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:51.689888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.689750 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-certs\") pod \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\" (UID: \"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6\") " Oct 30 18:39:51.690183 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.690155 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-data" pod "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" (UID: "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.690413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.690368 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.690413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.690393 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.690413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.690407 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.690597 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.690371 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-certs" pod "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" (UID: "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.690694 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.690674 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-credential-socket" pod "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" (UID: "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.692422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.692396 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-podinfo" pod "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" (UID: "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:39:51.692510 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.692442 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-token" pod "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" (UID: "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:51.692795 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.692763 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-envoy" pod "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" (UID: "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:39:51.693199 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.693175 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-kube-api-access-4qrs6" pod "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" (UID: "3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6"). InnerVolumeSpecName "kube-api-access-4qrs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:51.791556 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.791522 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.791556 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.791557 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qrs6\" (UniqueName: \"kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-kube-api-access-4qrs6\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.791778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.791570 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.791778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.791579 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.791778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.791588 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:51.791778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:51.791597 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:52.077576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:52.077492 4883 generic.go:356] "Generic (PLEG): container finished" podID="3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" containerID="dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7" exitCode=0 Oct 30 18:39:52.077576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:52.077548 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" Oct 30 18:39:52.078118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:52.077578 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" event={"ID":"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6","Type":"ContainerDied","Data":"dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7"} Oct 30 18:39:52.078118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:52.077634 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9" event={"ID":"3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6","Type":"ContainerDied","Data":"4e3bb41b9d36d58786027dc13479e5969cc16db492f1a143ee384906570078ef"} Oct 30 18:39:52.078118 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:52.077655 4883 scope.go:122] "RemoveContainer" containerID="dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7" Oct 30 18:39:52.079384 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:52.079357 4883 generic.go:356] "Generic (PLEG): container finished" podID="4734f1ff-9784-404b-ab17-94075cdae588" containerID="f29cb08f4912f898b4beeee3f022830ea00682ea88685b3714cf452911aa0e97" exitCode=0 Oct 30 18:39:52.079530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:52.079384 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" event={"ID":"4734f1ff-9784-404b-ab17-94075cdae588","Type":"ContainerDied","Data":"f29cb08f4912f898b4beeee3f022830ea00682ea88685b3714cf452911aa0e97"} Oct 30 18:39:52.089350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:52.089327 4883 scope.go:122] "RemoveContainer" containerID="dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7" Oct 30 18:39:52.089624 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:39:52.089603 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7\": container with ID starting with dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7 not found: ID does not exist" containerID="dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7" Oct 30 18:39:52.089677 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:52.089633 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7"} err="failed to get container status \"dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7\": rpc error: code = NotFound desc = could not find container \"dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7\": container with ID starting with dc273b9c62afbd067db25c3311fcdc7e64d208d28e3c715852734b9c02819de7 not found: ID does not exist" Oct 30 18:39:52.195244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:52.195208 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9"] Oct 30 18:39:52.207885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:52.207847 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e77b1d-openshift-default-c95557bdf-vbvh9"] Oct 30 18:39:53.180874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:53.180837 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" path="/var/lib/kubelet/pods/3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6/volumes" Oct 30 18:39:53.246596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:53.246563 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:53.412387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:53.412358 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4734f1ff-9784-404b-ab17-94075cdae588-kube-api-access-bnv7s\" (UniqueName: \"kubernetes.io/projected/4734f1ff-9784-404b-ab17-94075cdae588-kube-api-access-bnv7s\") pod \"4734f1ff-9784-404b-ab17-94075cdae588\" (UID: \"4734f1ff-9784-404b-ab17-94075cdae588\") " Oct 30 18:39:53.412585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:53.412401 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-xks-ca\") pod \"4734f1ff-9784-404b-ab17-94075cdae588\" (UID: \"4734f1ff-9784-404b-ab17-94075cdae588\") " Oct 30 18:39:53.412585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:53.412578 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-openshift-ca\") pod \"4734f1ff-9784-404b-ab17-94075cdae588\" (UID: \"4734f1ff-9784-404b-ab17-94075cdae588\") " Oct 30 18:39:53.413017 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:53.412985 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-openshift-ca" pod "4734f1ff-9784-404b-ab17-94075cdae588" (UID: "4734f1ff-9784-404b-ab17-94075cdae588"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:53.414494 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:53.414464 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-xks-ca" pod "4734f1ff-9784-404b-ab17-94075cdae588" (UID: "4734f1ff-9784-404b-ab17-94075cdae588"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:53.414596 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:53.414521 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4734f1ff-9784-404b-ab17-94075cdae588-kube-api-access-bnv7s" pod "4734f1ff-9784-404b-ab17-94075cdae588" (UID: "4734f1ff-9784-404b-ab17-94075cdae588"). InnerVolumeSpecName "kube-api-access-bnv7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:53.514608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:53.514577 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:53.514608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:53.514608 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnv7s\" (UniqueName: \"kubernetes.io/projected/4734f1ff-9784-404b-ab17-94075cdae588-kube-api-access-bnv7s\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:53.514813 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:53.514619 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4734f1ff-9784-404b-ab17-94075cdae588-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:39:54.094812 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:54.094778 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" Oct 30 18:39:54.095029 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:54.094812 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls" event={"ID":"4734f1ff-9784-404b-ab17-94075cdae588","Type":"ContainerDied","Data":"7b6ee89650000dc24e2b80ef661726e48a656761ab06ec7160c2916b78576744"} Oct 30 18:39:54.095029 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:39:54.094885 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b6ee89650000dc24e2b80ef661726e48a656761ab06ec7160c2916b78576744" Oct 30 18:40:03.333777 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:03.333735 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn"] Oct 30 18:40:03.367330 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:03.367275 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c8eef3-bpdrn"] Oct 30 18:40:04.615350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:04.615305 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:05.401961 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:40:05.376168 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:40:05.538960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:05.538104 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:05.612612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:05.612564 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:05.618197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:05.618146 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:05.633993 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:05.633776 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:05.642692 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:05.642295 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="861fdfb3-c9f1-4301-b316-3f5ddfa98496" path="/var/lib/kubelet/pods/861fdfb3-c9f1-4301-b316-3f5ddfa98496/volumes" Oct 30 18:40:05.683815 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:05.682310 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:40:06.080813 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:06.080727 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:40:09.028711 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:09.028676 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls"] Oct 30 18:40:09.038029 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:09.038004 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-3b8bf3-8kjls"] Oct 30 18:40:09.185702 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:09.185663 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4734f1ff-9784-404b-ab17-94075cdae588" path="/var/lib/kubelet/pods/4734f1ff-9784-404b-ab17-94075cdae588/volumes" Oct 30 18:40:11.236062 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:11.234335 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd"] Oct 30 18:40:11.236062 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:11.234625 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" containerID="cri-o://a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920" gracePeriod=30 Oct 30 18:40:12.050021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.049981 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4"] Oct 30 18:40:12.050726 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.050700 4883 cpu_manager.go:414] "containerMap: removing container" podUID="3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" containerName="istio-proxy" Oct 30 18:40:12.050726 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.050724 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" containerName="istio-proxy" Oct 30 18:40:12.050888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.050748 4883 cpu_manager.go:414] "containerMap: removing container" podUID="4734f1ff-9784-404b-ab17-94075cdae588" containerName="revoke-keys" Oct 30 18:40:12.050888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.050755 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4734f1ff-9784-404b-ab17-94075cdae588" containerName="revoke-keys" Oct 30 18:40:12.050888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.050810 4883 cpu_manager.go:414] "containerMap: removing container" podUID="861fdfb3-c9f1-4301-b316-3f5ddfa98496" containerName="revoke-keys" Oct 30 18:40:12.050888 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.050819 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="861fdfb3-c9f1-4301-b316-3f5ddfa98496" containerName="revoke-keys" Oct 30 18:40:12.051105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.050929 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="861fdfb3-c9f1-4301-b316-3f5ddfa98496" containerName="revoke-keys" Oct 30 18:40:12.051105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.050964 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7384b5-f1fc-40ea-9d0e-2ab3e19efdb6" containerName="istio-proxy" Oct 30 18:40:12.051105 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.050976 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="4734f1ff-9784-404b-ab17-94075cdae588" containerName="revoke-keys" Oct 30 18:40:12.056175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.056149 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:12.099471 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.099421 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:40:12.099680 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.099490 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:40:12.099680 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.099502 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:40:12.099845 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.099810 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:40:12.111417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.111383 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4"] Oct 30 18:40:12.159097 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.159061 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4\" (UID: \"4e86dbd2-1be5-44ba-9a26-eef77565a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:12.159295 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.159230 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4\" (UID: \"4e86dbd2-1be5-44ba-9a26-eef77565a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:12.159387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.159362 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfksq\" (UniqueName: \"kubernetes.io/projected/4e86dbd2-1be5-44ba-9a26-eef77565a328-kube-api-access-hfksq\") pod \"maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4\" (UID: \"4e86dbd2-1be5-44ba-9a26-eef77565a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:12.260604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.260564 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4\" (UID: \"4e86dbd2-1be5-44ba-9a26-eef77565a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:12.261177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.260683 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4\" (UID: \"4e86dbd2-1be5-44ba-9a26-eef77565a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:12.261177 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.260808 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfksq\" (UniqueName: \"kubernetes.io/projected/4e86dbd2-1be5-44ba-9a26-eef77565a328-kube-api-access-hfksq\") pod \"maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4\" (UID: \"4e86dbd2-1be5-44ba-9a26-eef77565a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:12.261778 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.261748 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4\" (UID: \"4e86dbd2-1be5-44ba-9a26-eef77565a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:12.263422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.263395 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4\" (UID: \"4e86dbd2-1be5-44ba-9a26-eef77565a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:12.285307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.285280 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfksq\" (UniqueName: \"kubernetes.io/projected/4e86dbd2-1be5-44ba-9a26-eef77565a328-kube-api-access-hfksq\") pod \"maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4\" (UID: \"4e86dbd2-1be5-44ba-9a26-eef77565a328\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:12.372564 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.372530 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:12.678848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.678818 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4"] Oct 30 18:40:12.681124 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:40:12.681081 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e86dbd2_1be5_44ba_9a26_eef77565a328.slice/crio-0f0e25d4377515120dd0a4b442bd13821e6865d750481d37d9578ebe8df29800 WatchSource:0}: Error finding container 0f0e25d4377515120dd0a4b442bd13821e6865d750481d37d9578ebe8df29800: Status 404 returned error can't find the container with id 0f0e25d4377515120dd0a4b442bd13821e6865d750481d37d9578ebe8df29800 Oct 30 18:40:12.691902 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:12.691875 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" event={"ID":"4e86dbd2-1be5-44ba-9a26-eef77565a328","Type":"ContainerStarted","Data":"0f0e25d4377515120dd0a4b442bd13821e6865d750481d37d9578ebe8df29800"} Oct 30 18:40:13.700300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:13.700212 4883 generic.go:356] "Generic (PLEG): container finished" podID="4e86dbd2-1be5-44ba-9a26-eef77565a328" containerID="6bdaeedbba98d19ccb1c7b48773c72f2447bc520928d5670f321453a5e9e8a5a" exitCode=0 Oct 30 18:40:13.700300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:13.700280 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" event={"ID":"4e86dbd2-1be5-44ba-9a26-eef77565a328","Type":"ContainerDied","Data":"6bdaeedbba98d19ccb1c7b48773c72f2447bc520928d5670f321453a5e9e8a5a"} Oct 30 18:40:14.892709 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:14.892682 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:14.993239 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:14.993161 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-openshift-ca\") pod \"4e86dbd2-1be5-44ba-9a26-eef77565a328\" (UID: \"4e86dbd2-1be5-44ba-9a26-eef77565a328\") " Oct 30 18:40:14.993421 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:14.993244 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e86dbd2-1be5-44ba-9a26-eef77565a328-kube-api-access-hfksq\" (UniqueName: \"kubernetes.io/projected/4e86dbd2-1be5-44ba-9a26-eef77565a328-kube-api-access-hfksq\") pod \"4e86dbd2-1be5-44ba-9a26-eef77565a328\" (UID: \"4e86dbd2-1be5-44ba-9a26-eef77565a328\") " Oct 30 18:40:14.993421 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:14.993309 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-xks-ca\") pod \"4e86dbd2-1be5-44ba-9a26-eef77565a328\" (UID: \"4e86dbd2-1be5-44ba-9a26-eef77565a328\") " Oct 30 18:40:14.993563 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:14.993534 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-openshift-ca" pod "4e86dbd2-1be5-44ba-9a26-eef77565a328" (UID: "4e86dbd2-1be5-44ba-9a26-eef77565a328"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:14.993813 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:14.993795 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:14.995358 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:14.995336 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e86dbd2-1be5-44ba-9a26-eef77565a328-kube-api-access-hfksq" pod "4e86dbd2-1be5-44ba-9a26-eef77565a328" (UID: "4e86dbd2-1be5-44ba-9a26-eef77565a328"). InnerVolumeSpecName "kube-api-access-hfksq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:14.995358 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:14.995343 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-xks-ca" pod "4e86dbd2-1be5-44ba-9a26-eef77565a328" (UID: "4e86dbd2-1be5-44ba-9a26-eef77565a328"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:15.094522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:15.094490 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfksq\" (UniqueName: \"kubernetes.io/projected/4e86dbd2-1be5-44ba-9a26-eef77565a328-kube-api-access-hfksq\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:15.094522 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:15.094519 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e86dbd2-1be5-44ba-9a26-eef77565a328-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:15.715545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:15.715516 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" Oct 30 18:40:15.715545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:15.715538 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4" event={"ID":"4e86dbd2-1be5-44ba-9a26-eef77565a328","Type":"ContainerDied","Data":"0f0e25d4377515120dd0a4b442bd13821e6865d750481d37d9578ebe8df29800"} Oct 30 18:40:15.715821 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:15.715586 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f0e25d4377515120dd0a4b442bd13821e6865d750481d37d9578ebe8df29800" Oct 30 18:40:15.757066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:15.757027 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:40:15.757244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:15.757096 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:40:16.503386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.503313 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:40:16.612682 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.612647 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-certs\") pod \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " Oct 30 18:40:16.612895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.612714 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-token\" (UniqueName: \"kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-token\") pod \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " Oct 30 18:40:16.612895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.612783 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-data\") pod \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " Oct 30 18:40:16.612895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.612845 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istiod-ca-cert\") pod \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " Oct 30 18:40:16.613089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.612912 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-envoy\") pod \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " Oct 30 18:40:16.613089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.612976 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-certs" pod "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" (UID: "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:16.613089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.613009 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-kube-api-access-x2nv7\" (UniqueName: \"kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-kube-api-access-x2nv7\") pod \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " Oct 30 18:40:16.613089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.613044 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-podinfo\") pod \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " Oct 30 18:40:16.613304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.613120 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-socket\") pod \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " Oct 30 18:40:16.613304 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.613185 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-credential-socket\") pod \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\" (UID: \"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d\") " Oct 30 18:40:16.613505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.613473 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-socket" pod "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" (UID: "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:16.613650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.613558 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-data" pod "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" (UID: "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:16.613755 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.613731 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-credential-socket" pod "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" (UID: "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:16.613842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.613763 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.613842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.613786 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.613842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.613796 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istiod-ca-cert" pod "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" (UID: "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:16.613842 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.613801 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.615517 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.615491 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-token" pod "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" (UID: "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:16.615711 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.615689 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-kube-api-access-x2nv7" pod "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" (UID: "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d"). InnerVolumeSpecName "kube-api-access-x2nv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:16.615917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.615895 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-envoy" pod "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" (UID: "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:16.616137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.616113 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-podinfo" pod "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" (UID: "5c5af977-f5ee-480f-bcc1-8c4a5ff5257d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:16.715369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.715331 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.715369 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.715366 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.715598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.715379 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2nv7\" (UniqueName: \"kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-kube-api-access-x2nv7\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.715598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.715394 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.715598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.715407 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.715598 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.715417 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:16.724219 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.724185 4883 generic.go:356] "Generic (PLEG): container finished" podID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerID="a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920" exitCode=0 Oct 30 18:40:16.724373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.724236 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" Oct 30 18:40:16.724373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.724267 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" event={"ID":"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d","Type":"ContainerDied","Data":"a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920"} Oct 30 18:40:16.724373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.724343 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd" event={"ID":"5c5af977-f5ee-480f-bcc1-8c4a5ff5257d","Type":"ContainerDied","Data":"6c4bfb77bab844335cf116a94708f9470ca53626cd363b39c31f2e96e4461954"} Oct 30 18:40:16.724373 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.724366 4883 scope.go:122] "RemoveContainer" containerID="a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920" Oct 30 18:40:16.736293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.736266 4883 scope.go:122] "RemoveContainer" containerID="42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3" Oct 30 18:40:16.748125 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.748101 4883 scope.go:122] "RemoveContainer" containerID="a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920" Oct 30 18:40:16.748455 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:40:16.748434 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920\": container with ID starting with a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920 not found: ID does not exist" containerID="a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920" Oct 30 18:40:16.748508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.748467 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920"} err="failed to get container status \"a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920\": rpc error: code = NotFound desc = could not find container \"a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920\": container with ID starting with a5544ba85b09363df37067bbea88664f9ef7cda7b7707a9bd03d7327ef226920 not found: ID does not exist" Oct 30 18:40:16.748508 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.748486 4883 scope.go:122] "RemoveContainer" containerID="42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3" Oct 30 18:40:16.748764 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:40:16.748749 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3\": container with ID starting with 42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3 not found: ID does not exist" containerID="42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3" Oct 30 18:40:16.748826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.748766 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3"} err="failed to get container status \"42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3\": rpc error: code = NotFound desc = could not find container \"42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3\": container with ID starting with 42ee9ad588200fd2c98aec6c73a5c484c31b6e4c7c812ea75a3a4e9142bd8cc3 not found: ID does not exist" Oct 30 18:40:16.850481 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.850442 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd"] Oct 30 18:40:16.853175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.853147 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm"] Oct 30 18:40:16.856635 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.856607 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c8eef3-openshift-default-64dd957bff-jdkkd"] Oct 30 18:40:16.906163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:16.906133 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj"] Oct 30 18:40:17.178886 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:17.178813 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" path="/var/lib/kubelet/pods/5c5af977-f5ee-480f-bcc1-8c4a5ff5257d/volumes" Oct 30 18:40:18.170783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.170748 4883 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" secret="" err="secret \"e2e-shared-b-w4-3b8bf3-openshift-default-dockercfg-256pt\" not found" Oct 30 18:40:18.178612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.178579 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx"] Oct 30 18:40:18.740743 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.740703 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" podUID="3ecdc930-9541-4a66-9967-4fb76ca616d9" containerName="istio-proxy" containerID="cri-o://f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753" gracePeriod=30 Oct 30 18:40:18.958486 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.958456 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw"] Oct 30 18:40:18.958982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.958969 4883 cpu_manager.go:414] "containerMap: removing container" podUID="4e86dbd2-1be5-44ba-9a26-eef77565a328" containerName="revoke-keys" Oct 30 18:40:18.959047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.958983 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e86dbd2-1be5-44ba-9a26-eef77565a328" containerName="revoke-keys" Oct 30 18:40:18.959047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.958998 4883 cpu_manager.go:414] "containerMap: removing container" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" Oct 30 18:40:18.959047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.959003 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" Oct 30 18:40:18.959047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.959019 4883 cpu_manager.go:414] "containerMap: removing container" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" Oct 30 18:40:18.959047 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.959025 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" Oct 30 18:40:18.959216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.959101 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e86dbd2-1be5-44ba-9a26-eef77565a328" containerName="revoke-keys" Oct 30 18:40:18.959216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.959109 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" Oct 30 18:40:18.963987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.963953 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:18.984321 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.984289 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:40:18.984502 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.984396 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:40:18.984565 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.984526 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:40:18.984639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.984600 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:40:18.987625 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:18.987597 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw"] Oct 30 18:40:19.149644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:19.149593 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2crt\" (UniqueName: \"kubernetes.io/projected/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-kube-api-access-s2crt\") pod \"maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw\" (UID: \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:19.149851 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:19.149834 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw\" (UID: \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:19.149895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:19.149875 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw\" (UID: \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:19.251471 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:19.251430 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw\" (UID: \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:19.251977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:19.251479 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw\" (UID: \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:19.251977 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:19.251587 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2crt\" (UniqueName: \"kubernetes.io/projected/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-kube-api-access-s2crt\") pod \"maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw\" (UID: \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:19.252214 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:19.252192 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw\" (UID: \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:19.254044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:19.254019 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw\" (UID: \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:19.280823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:19.280791 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2crt\" (UniqueName: \"kubernetes.io/projected/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-kube-api-access-s2crt\") pod \"maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw\" (UID: \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:19.579824 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:19.579733 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:19.897878 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:19.897848 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw"] Oct 30 18:40:19.900038 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:40:19.900003 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00efc6b_3f0f_4e79_8a99_21be3e64a24d.slice/crio-36885f95a7ca4ee05704403964b1811b4365aa5476390286b3752bd8caaecd80 WatchSource:0}: Error finding container 36885f95a7ca4ee05704403964b1811b4365aa5476390286b3752bd8caaecd80: Status 404 returned error can't find the container with id 36885f95a7ca4ee05704403964b1811b4365aa5476390286b3752bd8caaecd80 Oct 30 18:40:20.755215 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:20.755177 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" event={"ID":"a00efc6b-3f0f-4e79-8a99-21be3e64a24d","Type":"ContainerStarted","Data":"8f14abc3c0ec58bb095a90334454460c0ca8d5316922be0ef96a5fb28b9d41a4"} Oct 30 18:40:20.755644 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:20.755238 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" event={"ID":"a00efc6b-3f0f-4e79-8a99-21be3e64a24d","Type":"ContainerStarted","Data":"36885f95a7ca4ee05704403964b1811b4365aa5476390286b3752bd8caaecd80"} Oct 30 18:40:20.799353 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:20.799297 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" podStartSLOduration=2.799277862 podStartE2EDuration="2.799277862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:20.797824935 +0000 UTC m=+2879.754828770" watchObservedRunningTime="2026-10-30 18:40:20.799277862 +0000 UTC m=+2879.756281706" Oct 30 18:40:21.763334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:21.763297 4883 generic.go:356] "Generic (PLEG): container finished" podID="a00efc6b-3f0f-4e79-8a99-21be3e64a24d" containerID="8f14abc3c0ec58bb095a90334454460c0ca8d5316922be0ef96a5fb28b9d41a4" exitCode=0 Oct 30 18:40:21.763760 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:21.763377 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" event={"ID":"a00efc6b-3f0f-4e79-8a99-21be3e64a24d","Type":"ContainerDied","Data":"8f14abc3c0ec58bb095a90334454460c0ca8d5316922be0ef96a5fb28b9d41a4"} Oct 30 18:40:22.956157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:22.956129 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:23.014237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.013920 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-openshift-ca\") pod \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\" (UID: \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\") " Oct 30 18:40:23.014237 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.014030 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-kube-api-access-s2crt\" (UniqueName: \"kubernetes.io/projected/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-kube-api-access-s2crt\") pod \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\" (UID: \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\") " Oct 30 18:40:23.014489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.014253 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-xks-ca\") pod \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\" (UID: \"a00efc6b-3f0f-4e79-8a99-21be3e64a24d\") " Oct 30 18:40:23.014489 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.014342 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-openshift-ca" pod "a00efc6b-3f0f-4e79-8a99-21be3e64a24d" (UID: "a00efc6b-3f0f-4e79-8a99-21be3e64a24d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:23.014929 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.014903 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.017297 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.017268 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-xks-ca" pod "a00efc6b-3f0f-4e79-8a99-21be3e64a24d" (UID: "a00efc6b-3f0f-4e79-8a99-21be3e64a24d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:23.017470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.017409 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-kube-api-access-s2crt" pod "a00efc6b-3f0f-4e79-8a99-21be3e64a24d" (UID: "a00efc6b-3f0f-4e79-8a99-21be3e64a24d"). InnerVolumeSpecName "kube-api-access-s2crt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:23.115638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.115600 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.115827 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.115661 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2crt\" (UniqueName: \"kubernetes.io/projected/a00efc6b-3f0f-4e79-8a99-21be3e64a24d-kube-api-access-s2crt\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.554667 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.554626 4883 patch_prober.go:29] interesting pod/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:40:23.554871 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.554698 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" podUID="3ecdc930-9541-4a66-9967-4fb76ca616d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:40:23.655759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.655719 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr"] Oct 30 18:40:23.677276 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.669428 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8"] Oct 30 18:40:23.682279 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.682247 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns"] Oct 30 18:40:23.781893 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.781860 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" event={"ID":"a00efc6b-3f0f-4e79-8a99-21be3e64a24d","Type":"ContainerDied","Data":"36885f95a7ca4ee05704403964b1811b4365aa5476390286b3752bd8caaecd80"} Oct 30 18:40:23.781893 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.781901 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw" Oct 30 18:40:23.782104 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:23.781916 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36885f95a7ca4ee05704403964b1811b4365aa5476390286b3752bd8caaecd80" Oct 30 18:40:24.031337 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.031308 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:40:24.131741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.131646 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-socket\") pod \"3ecdc930-9541-4a66-9967-4fb76ca616d9\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " Oct 30 18:40:24.131741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.131703 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-podinfo\") pod \"3ecdc930-9541-4a66-9967-4fb76ca616d9\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " Oct 30 18:40:24.131741 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.131731 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-credential-socket\") pod \"3ecdc930-9541-4a66-9967-4fb76ca616d9\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " Oct 30 18:40:24.132076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.131757 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-data\") pod \"3ecdc930-9541-4a66-9967-4fb76ca616d9\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " Oct 30 18:40:24.132076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.131796 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-kube-api-access-sdxhc\" (UniqueName: \"kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-kube-api-access-sdxhc\") pod \"3ecdc930-9541-4a66-9967-4fb76ca616d9\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " Oct 30 18:40:24.132076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.131872 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-envoy\") pod \"3ecdc930-9541-4a66-9967-4fb76ca616d9\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " Oct 30 18:40:24.132076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.131954 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-certs\") pod \"3ecdc930-9541-4a66-9967-4fb76ca616d9\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " Oct 30 18:40:24.132076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.131992 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ecdc930-9541-4a66-9967-4fb76ca616d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecdc930-9541-4a66-9967-4fb76ca616d9-istiod-ca-cert\") pod \"3ecdc930-9541-4a66-9967-4fb76ca616d9\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " Oct 30 18:40:24.132324 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.132096 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-token\" (UniqueName: \"kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-token\") pod \"3ecdc930-9541-4a66-9967-4fb76ca616d9\" (UID: \"3ecdc930-9541-4a66-9967-4fb76ca616d9\") " Oct 30 18:40:24.132837 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.132804 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-socket" pod "3ecdc930-9541-4a66-9967-4fb76ca616d9" (UID: "3ecdc930-9541-4a66-9967-4fb76ca616d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:24.133070 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.133046 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-certs" pod "3ecdc930-9541-4a66-9967-4fb76ca616d9" (UID: "3ecdc930-9541-4a66-9967-4fb76ca616d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:24.133658 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.133595 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-credential-socket" pod "3ecdc930-9541-4a66-9967-4fb76ca616d9" (UID: "3ecdc930-9541-4a66-9967-4fb76ca616d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:24.134225 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.134168 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-data" pod "3ecdc930-9541-4a66-9967-4fb76ca616d9" (UID: "3ecdc930-9541-4a66-9967-4fb76ca616d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:24.134772 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.134748 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecdc930-9541-4a66-9967-4fb76ca616d9-istiod-ca-cert" pod "3ecdc930-9541-4a66-9967-4fb76ca616d9" (UID: "3ecdc930-9541-4a66-9967-4fb76ca616d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:24.135970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.135924 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-token" pod "3ecdc930-9541-4a66-9967-4fb76ca616d9" (UID: "3ecdc930-9541-4a66-9967-4fb76ca616d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:24.136256 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.136238 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-envoy" pod "3ecdc930-9541-4a66-9967-4fb76ca616d9" (UID: "3ecdc930-9541-4a66-9967-4fb76ca616d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:24.136314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.136291 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-podinfo" pod "3ecdc930-9541-4a66-9967-4fb76ca616d9" (UID: "3ecdc930-9541-4a66-9967-4fb76ca616d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:24.136996 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.136969 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-kube-api-access-sdxhc" pod "3ecdc930-9541-4a66-9967-4fb76ca616d9" (UID: "3ecdc930-9541-4a66-9967-4fb76ca616d9"). InnerVolumeSpecName "kube-api-access-sdxhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:24.233466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.233423 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:24.233466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.233457 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:24.233466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.233470 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:24.233739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.233481 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:24.233739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.233496 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdxhc\" (UniqueName: \"kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-kube-api-access-sdxhc\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:24.233739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.233507 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:24.233739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.233518 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ecdc930-9541-4a66-9967-4fb76ca616d9-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:24.233739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.233531 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ecdc930-9541-4a66-9967-4fb76ca616d9-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:24.233739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.233541 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ecdc930-9541-4a66-9967-4fb76ca616d9-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:24.790440 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.790404 4883 generic.go:356] "Generic (PLEG): container finished" podID="3ecdc930-9541-4a66-9967-4fb76ca616d9" containerID="f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753" exitCode=0 Oct 30 18:40:24.790599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.790448 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" Oct 30 18:40:24.790599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.790484 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" event={"ID":"3ecdc930-9541-4a66-9967-4fb76ca616d9","Type":"ContainerDied","Data":"f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753"} Oct 30 18:40:24.790599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.790542 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx" event={"ID":"3ecdc930-9541-4a66-9967-4fb76ca616d9","Type":"ContainerDied","Data":"63bcd9550e4decbee141b558f4649910b24beec1725c52c8a2e6705bc8321a22"} Oct 30 18:40:24.790599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.790569 4883 scope.go:122] "RemoveContainer" containerID="f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753" Oct 30 18:40:24.802227 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.802206 4883 scope.go:122] "RemoveContainer" containerID="f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753" Oct 30 18:40:24.802551 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:40:24.802534 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753\": container with ID starting with f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753 not found: ID does not exist" containerID="f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753" Oct 30 18:40:24.802600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.802561 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753"} err="failed to get container status \"f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753\": rpc error: code = NotFound desc = could not find container \"f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753\": container with ID starting with f2f50f5be5fba297af4ab37b6e782b3ea6e041de207fbf494c9d7da294529753 not found: ID does not exist" Oct 30 18:40:24.865490 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.865438 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx"] Oct 30 18:40:24.878708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:24.878678 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-3b8bf3-openshift-default-69884564c7-mqfhx"] Oct 30 18:40:25.177443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:25.177409 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ecdc930-9541-4a66-9967-4fb76ca616d9" path="/var/lib/kubelet/pods/3ecdc930-9541-4a66-9967-4fb76ca616d9/volumes" Oct 30 18:40:28.778181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.778142 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5"] Oct 30 18:40:28.778631 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.778617 4883 cpu_manager.go:414] "containerMap: removing container" podUID="a00efc6b-3f0f-4e79-8a99-21be3e64a24d" containerName="revoke-keys" Oct 30 18:40:28.778680 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.778633 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00efc6b-3f0f-4e79-8a99-21be3e64a24d" containerName="revoke-keys" Oct 30 18:40:28.778717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.778681 4883 cpu_manager.go:414] "containerMap: removing container" podUID="3ecdc930-9541-4a66-9967-4fb76ca616d9" containerName="istio-proxy" Oct 30 18:40:28.778717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.778687 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ecdc930-9541-4a66-9967-4fb76ca616d9" containerName="istio-proxy" Oct 30 18:40:28.778781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.778768 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ecdc930-9541-4a66-9967-4fb76ca616d9" containerName="istio-proxy" Oct 30 18:40:28.778781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.778779 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00efc6b-3f0f-4e79-8a99-21be3e64a24d" containerName="revoke-keys" Oct 30 18:40:28.778858 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.778787 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5af977-f5ee-480f-bcc1-8c4a5ff5257d" containerName="istio-proxy" Oct 30 18:40:28.783535 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.783511 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:28.800773 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.800742 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj"] Oct 30 18:40:28.804880 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.804856 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:28.811388 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.811362 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:40:28.811638 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.811573 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:40:28.820715 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.818186 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:40:28.820715 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.818281 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:40:28.821715 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.821501 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5"] Oct 30 18:40:28.840915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.840874 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj"] Oct 30 18:40:28.891078 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.891001 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-066e3de6-8klmj\" (UID: \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:28.891272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.891158 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpl4t\" (UniqueName: \"kubernetes.io/projected/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-kube-api-access-rpl4t\") pod \"maas-api-revoke-sub-066e3de6-8klmj\" (UID: \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:28.891272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.891244 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6da30caa-2phj5\" (UID: \"211f0fa1-8896-4b7c-804b-9909d475c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:28.891430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.891406 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6da30caa-2phj5\" (UID: \"211f0fa1-8896-4b7c-804b-9909d475c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:28.891491 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.891445 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htf8c\" (UniqueName: \"kubernetes.io/projected/211f0fa1-8896-4b7c-804b-9909d475c6e2-kube-api-access-htf8c\") pod \"maas-api-revoke-sub-6da30caa-2phj5\" (UID: \"211f0fa1-8896-4b7c-804b-9909d475c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:28.891491 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.891471 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-066e3de6-8klmj\" (UID: \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:28.992394 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.992357 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-066e3de6-8klmj\" (UID: \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:28.992569 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.992411 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-066e3de6-8klmj\" (UID: \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:28.992569 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.992464 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpl4t\" (UniqueName: \"kubernetes.io/projected/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-kube-api-access-rpl4t\") pod \"maas-api-revoke-sub-066e3de6-8klmj\" (UID: \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:28.992569 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.992527 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6da30caa-2phj5\" (UID: \"211f0fa1-8896-4b7c-804b-9909d475c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:28.992816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.992623 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6da30caa-2phj5\" (UID: \"211f0fa1-8896-4b7c-804b-9909d475c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:28.992816 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.992654 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htf8c\" (UniqueName: \"kubernetes.io/projected/211f0fa1-8896-4b7c-804b-9909d475c6e2-kube-api-access-htf8c\") pod \"maas-api-revoke-sub-6da30caa-2phj5\" (UID: \"211f0fa1-8896-4b7c-804b-9909d475c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:28.993211 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.993185 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-066e3de6-8klmj\" (UID: \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:28.993298 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.993267 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6da30caa-2phj5\" (UID: \"211f0fa1-8896-4b7c-804b-9909d475c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:28.994906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.994885 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-066e3de6-8klmj\" (UID: \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:28.995078 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:28.995060 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6da30caa-2phj5\" (UID: \"211f0fa1-8896-4b7c-804b-9909d475c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:29.039889 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.039817 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htf8c\" (UniqueName: \"kubernetes.io/projected/211f0fa1-8896-4b7c-804b-9909d475c6e2-kube-api-access-htf8c\") pod \"maas-api-revoke-sub-6da30caa-2phj5\" (UID: \"211f0fa1-8896-4b7c-804b-9909d475c6e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:29.040030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.039960 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpl4t\" (UniqueName: \"kubernetes.io/projected/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-kube-api-access-rpl4t\") pod \"maas-api-revoke-sub-066e3de6-8klmj\" (UID: \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:29.098009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.097978 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:29.126286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.126254 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6"] Oct 30 18:40:29.126428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.126352 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:29.131684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.131659 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.150597 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.150559 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ac422f-openshift-default-dockercfg-w59vs\"" Oct 30 18:40:29.156989 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.156068 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6"] Oct 30 18:40:29.195717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.195681 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-data\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.195869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.195723 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-certs\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.195869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.195748 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-credential-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.195869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.195806 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4n8s\" (UniqueName: \"kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-kube-api-access-f4n8s\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.197005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.196977 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.197127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.197039 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-envoy\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.197127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.197089 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-podinfo\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.197127 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.197117 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-token\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.197294 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.197158 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istiod-ca-cert\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.305289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.301873 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4n8s\" (UniqueName: \"kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-kube-api-access-f4n8s\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.305289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.302011 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.305289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.302059 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-envoy\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.305289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.302107 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-podinfo\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.305289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.302136 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-token\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.305289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.302180 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istiod-ca-cert\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.305289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.302300 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-data\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.305289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.302345 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-certs\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.305289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.302386 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-credential-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.305289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.303000 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-credential-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.305289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.303868 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-socket\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.305289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.304928 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-data\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.306077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.305305 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-certs\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.306355 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.306304 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istiod-ca-cert\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.308383 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.308350 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-envoy\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.309044 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.309004 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-podinfo\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.327849 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.327817 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-token\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.328021 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.327962 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4n8s\" (UniqueName: \"kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-kube-api-access-f4n8s\") pod \"e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.414883 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.414846 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5"] Oct 30 18:40:29.419329 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:40:29.419039 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211f0fa1_8896_4b7c_804b_9909d475c6e2.slice/crio-2a4a09839eebc7328d237cbc8d4d68b8323efffd2a98b9cacfa1031fd508a291 WatchSource:0}: Error finding container 2a4a09839eebc7328d237cbc8d4d68b8323efffd2a98b9cacfa1031fd508a291: Status 404 returned error can't find the container with id 2a4a09839eebc7328d237cbc8d4d68b8323efffd2a98b9cacfa1031fd508a291 Oct 30 18:40:29.449723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.449695 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:29.452891 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.452862 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj"] Oct 30 18:40:29.455308 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:40:29.455287 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae7a355_7255_4fcf_b5dd_5bce0659f0df.slice/crio-5302ad83da1e79ef9e5a420dc6cc4758b9215dc4949c28e5b030889c1c577b2a WatchSource:0}: Error finding container 5302ad83da1e79ef9e5a420dc6cc4758b9215dc4949c28e5b030889c1c577b2a: Status 404 returned error can't find the container with id 5302ad83da1e79ef9e5a420dc6cc4758b9215dc4949c28e5b030889c1c577b2a Oct 30 18:40:29.833229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.833126 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" event={"ID":"5ae7a355-7255-4fcf-b5dd-5bce0659f0df","Type":"ContainerStarted","Data":"5302ad83da1e79ef9e5a420dc6cc4758b9215dc4949c28e5b030889c1c577b2a"} Oct 30 18:40:29.834910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.834882 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" event={"ID":"211f0fa1-8896-4b7c-804b-9909d475c6e2","Type":"ContainerStarted","Data":"5b591d631795c86584ea55373f22cc1df44689810642fe025ff2df716ac10a2c"} Oct 30 18:40:29.835113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.834943 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" event={"ID":"211f0fa1-8896-4b7c-804b-9909d475c6e2","Type":"ContainerStarted","Data":"2a4a09839eebc7328d237cbc8d4d68b8323efffd2a98b9cacfa1031fd508a291"} Oct 30 18:40:29.879264 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.879209 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" podStartSLOduration=1.879192355 podStartE2EDuration="1.879192355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:29.877158362 +0000 UTC m=+2888.834162180" watchObservedRunningTime="2026-10-30 18:40:29.879192355 +0000 UTC m=+2888.836196192" Oct 30 18:40:29.884904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.884872 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6"] Oct 30 18:40:29.886727 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:40:29.886692 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a60d9c1_369e_4818_8e6b_80e5bd537d36.slice/crio-76c0dcff6a088f26b33fd789b6bf8f68d98a5022c6d00d893a394297642bde43 WatchSource:0}: Error finding container 76c0dcff6a088f26b33fd789b6bf8f68d98a5022c6d00d893a394297642bde43: Status 404 returned error can't find the container with id 76c0dcff6a088f26b33fd789b6bf8f68d98a5022c6d00d893a394297642bde43 Oct 30 18:40:29.889077 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.889045 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:40:29.889156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.889109 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:40:29.889156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:29.889142 4883 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:40:30.851952 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:30.851898 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" event={"ID":"5a60d9c1-369e-4818-8e6b-80e5bd537d36","Type":"ContainerStarted","Data":"6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d"} Oct 30 18:40:30.852422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:30.852014 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" event={"ID":"5a60d9c1-369e-4818-8e6b-80e5bd537d36","Type":"ContainerStarted","Data":"76c0dcff6a088f26b33fd789b6bf8f68d98a5022c6d00d893a394297642bde43"} Oct 30 18:40:30.853751 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:30.853723 4883 generic.go:356] "Generic (PLEG): container finished" podID="5ae7a355-7255-4fcf-b5dd-5bce0659f0df" containerID="f01d08236981a8eb81de552c08e4f8f65acd25bea04874ea21a55b2a07eab0e2" exitCode=0 Oct 30 18:40:30.853869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:30.853808 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" event={"ID":"5ae7a355-7255-4fcf-b5dd-5bce0659f0df","Type":"ContainerDied","Data":"f01d08236981a8eb81de552c08e4f8f65acd25bea04874ea21a55b2a07eab0e2"} Oct 30 18:40:30.855367 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:30.855349 4883 generic.go:356] "Generic (PLEG): container finished" podID="211f0fa1-8896-4b7c-804b-9909d475c6e2" containerID="5b591d631795c86584ea55373f22cc1df44689810642fe025ff2df716ac10a2c" exitCode=0 Oct 30 18:40:30.855476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:30.855429 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" event={"ID":"211f0fa1-8896-4b7c-804b-9909d475c6e2","Type":"ContainerDied","Data":"5b591d631795c86584ea55373f22cc1df44689810642fe025ff2df716ac10a2c"} Oct 30 18:40:30.917034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:30.916959 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" podStartSLOduration=1.916913654 podStartE2EDuration="1.916913654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:30.91644471 +0000 UTC m=+2889.873448541" watchObservedRunningTime="2026-10-30 18:40:30.916913654 +0000 UTC m=+2889.873917487" Oct 30 18:40:31.450479 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:31.450446 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:31.452079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:31.452052 4883 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:31.452192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:31.452113 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" podUID="5a60d9c1-369e-4818-8e6b-80e5bd537d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 18:40:31.890516 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:31.890469 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" containerID="cri-o://c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272" gracePeriod=45 Oct 30 18:40:31.927733 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:31.927511 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" containerID="cri-o://dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43" gracePeriod=45 Oct 30 18:40:32.121547 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.120879 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:32.130251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.130223 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:32.261164 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.261138 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:40:32.295971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.294626 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-xks-ca\") pod \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\" (UID: \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\") " Oct 30 18:40:32.295971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.295128 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-kube-api-access-rpl4t\" (UniqueName: \"kubernetes.io/projected/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-kube-api-access-rpl4t\") pod \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\" (UID: \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\") " Oct 30 18:40:32.295971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.295707 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-openshift-ca\") pod \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\" (UID: \"5ae7a355-7255-4fcf-b5dd-5bce0659f0df\") " Oct 30 18:40:32.296226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.296121 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-openshift-ca" pod "5ae7a355-7255-4fcf-b5dd-5bce0659f0df" (UID: "5ae7a355-7255-4fcf-b5dd-5bce0659f0df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:32.296278 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.296255 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-openshift-ca\") pod \"211f0fa1-8896-4b7c-804b-9909d475c6e2\" (UID: \"211f0fa1-8896-4b7c-804b-9909d475c6e2\") " Oct 30 18:40:32.296358 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.296306 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-xks-ca\") pod \"211f0fa1-8896-4b7c-804b-9909d475c6e2\" (UID: \"211f0fa1-8896-4b7c-804b-9909d475c6e2\") " Oct 30 18:40:32.296358 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.296343 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211f0fa1-8896-4b7c-804b-9909d475c6e2-kube-api-access-htf8c\" (UniqueName: \"kubernetes.io/projected/211f0fa1-8896-4b7c-804b-9909d475c6e2-kube-api-access-htf8c\") pod \"211f0fa1-8896-4b7c-804b-9909d475c6e2\" (UID: \"211f0fa1-8896-4b7c-804b-9909d475c6e2\") " Oct 30 18:40:32.298265 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.298138 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-kube-api-access-rpl4t" pod "5ae7a355-7255-4fcf-b5dd-5bce0659f0df" (UID: "5ae7a355-7255-4fcf-b5dd-5bce0659f0df"). InnerVolumeSpecName "kube-api-access-rpl4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:32.298346 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.298285 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-openshift-ca" pod "211f0fa1-8896-4b7c-804b-9909d475c6e2" (UID: "211f0fa1-8896-4b7c-804b-9909d475c6e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:32.298526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.298417 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpl4t\" (UniqueName: \"kubernetes.io/projected/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-kube-api-access-rpl4t\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.298526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.298439 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.298526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.298455 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.310323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.310290 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-xks-ca" pod "211f0fa1-8896-4b7c-804b-9909d475c6e2" (UID: "211f0fa1-8896-4b7c-804b-9909d475c6e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:32.310323 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.310301 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-xks-ca" pod "5ae7a355-7255-4fcf-b5dd-5bce0659f0df" (UID: "5ae7a355-7255-4fcf-b5dd-5bce0659f0df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:32.310535 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.310502 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:40:32.311557 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.311531 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211f0fa1-8896-4b7c-804b-9909d475c6e2-kube-api-access-htf8c" pod "211f0fa1-8896-4b7c-804b-9909d475c6e2" (UID: "211f0fa1-8896-4b7c-804b-9909d475c6e2"). InnerVolumeSpecName "kube-api-access-htf8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:32.399550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.399450 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-dshm\") pod \"c49b507f-a09f-4f74-8446-1cfc76f2418e\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " Oct 30 18:40:32.399550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.399527 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-tmp-dir\") pod \"c49b507f-a09f-4f74-8446-1cfc76f2418e\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " Oct 30 18:40:32.399789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.399594 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c49b507f-a09f-4f74-8446-1cfc76f2418e-tls-certs\" (UniqueName: \"kubernetes.io/secret/c49b507f-a09f-4f74-8446-1cfc76f2418e-tls-certs\") pod \"c49b507f-a09f-4f74-8446-1cfc76f2418e\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " Oct 30 18:40:32.399789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.399670 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-dshm\") pod \"cf31407f-9393-4e29-abd5-11332bf68e81\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " Oct 30 18:40:32.399789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.399719 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49b507f-a09f-4f74-8446-1cfc76f2418e-kube-api-access-kchwc\" (UniqueName: \"kubernetes.io/projected/c49b507f-a09f-4f74-8446-1cfc76f2418e-kube-api-access-kchwc\") pod \"c49b507f-a09f-4f74-8446-1cfc76f2418e\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " Oct 30 18:40:32.399789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.399756 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-model-cache\") pod \"c49b507f-a09f-4f74-8446-1cfc76f2418e\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " Oct 30 18:40:32.400046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.399812 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-tmp-dir\") pod \"cf31407f-9393-4e29-abd5-11332bf68e81\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " Oct 30 18:40:32.400046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.399842 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf31407f-9393-4e29-abd5-11332bf68e81-kube-api-access-zz7wl\" (UniqueName: \"kubernetes.io/projected/cf31407f-9393-4e29-abd5-11332bf68e81-kube-api-access-zz7wl\") pod \"cf31407f-9393-4e29-abd5-11332bf68e81\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " Oct 30 18:40:32.400046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.399968 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-home\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-home\") pod \"c49b507f-a09f-4f74-8446-1cfc76f2418e\" (UID: \"c49b507f-a09f-4f74-8446-1cfc76f2418e\") " Oct 30 18:40:32.400046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.399997 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-home\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-home\") pod \"cf31407f-9393-4e29-abd5-11332bf68e81\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " Oct 30 18:40:32.400258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.400051 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf31407f-9393-4e29-abd5-11332bf68e81-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf31407f-9393-4e29-abd5-11332bf68e81-tls-certs\") pod \"cf31407f-9393-4e29-abd5-11332bf68e81\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " Oct 30 18:40:32.400258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.400096 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-model-cache\") pod \"cf31407f-9393-4e29-abd5-11332bf68e81\" (UID: \"cf31407f-9393-4e29-abd5-11332bf68e81\") " Oct 30 18:40:32.400258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.400099 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-model-cache" pod "c49b507f-a09f-4f74-8446-1cfc76f2418e" (UID: "c49b507f-a09f-4f74-8446-1cfc76f2418e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.400258 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.400116 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-tmp-dir" pod "c49b507f-a09f-4f74-8446-1cfc76f2418e" (UID: "c49b507f-a09f-4f74-8446-1cfc76f2418e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.400653 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.400616 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-home" pod "cf31407f-9393-4e29-abd5-11332bf68e81" (UID: "cf31407f-9393-4e29-abd5-11332bf68e81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.400653 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.400627 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.400844 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.400667 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.400844 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.400683 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211f0fa1-8896-4b7c-804b-9909d475c6e2-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.400844 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.400698 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htf8c\" (UniqueName: \"kubernetes.io/projected/211f0fa1-8896-4b7c-804b-9909d475c6e2-kube-api-access-htf8c\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.400844 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.400712 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae7a355-7255-4fcf-b5dd-5bce0659f0df-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.401197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.400845 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-model-cache" pod "cf31407f-9393-4e29-abd5-11332bf68e81" (UID: "cf31407f-9393-4e29-abd5-11332bf68e81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.401197 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.401090 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-tmp-dir" pod "cf31407f-9393-4e29-abd5-11332bf68e81" (UID: "cf31407f-9393-4e29-abd5-11332bf68e81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.401359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.401242 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-home" pod "c49b507f-a09f-4f74-8446-1cfc76f2418e" (UID: "c49b507f-a09f-4f74-8446-1cfc76f2418e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.403466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.403438 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf31407f-9393-4e29-abd5-11332bf68e81-tls-certs" pod "cf31407f-9393-4e29-abd5-11332bf68e81" (UID: "cf31407f-9393-4e29-abd5-11332bf68e81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:32.405791 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.404728 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf31407f-9393-4e29-abd5-11332bf68e81-kube-api-access-zz7wl" pod "cf31407f-9393-4e29-abd5-11332bf68e81" (UID: "cf31407f-9393-4e29-abd5-11332bf68e81"). InnerVolumeSpecName "kube-api-access-zz7wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:32.406465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.406440 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c49b507f-a09f-4f74-8446-1cfc76f2418e-tls-certs" pod "c49b507f-a09f-4f74-8446-1cfc76f2418e" (UID: "c49b507f-a09f-4f74-8446-1cfc76f2418e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:32.406684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.406560 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-dshm" pod "c49b507f-a09f-4f74-8446-1cfc76f2418e" (UID: "c49b507f-a09f-4f74-8446-1cfc76f2418e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.406684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.406654 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-dshm" pod "cf31407f-9393-4e29-abd5-11332bf68e81" (UID: "cf31407f-9393-4e29-abd5-11332bf68e81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:32.407904 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.407878 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49b507f-a09f-4f74-8446-1cfc76f2418e-kube-api-access-kchwc" pod "c49b507f-a09f-4f74-8446-1cfc76f2418e" (UID: "c49b507f-a09f-4f74-8446-1cfc76f2418e"). InnerVolumeSpecName "kube-api-access-kchwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:32.450312 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.450277 4883 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:32.450465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.450331 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" podUID="5a60d9c1-369e-4818-8e6b-80e5bd537d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 18:40:32.501781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.501748 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kchwc\" (UniqueName: \"kubernetes.io/projected/c49b507f-a09f-4f74-8446-1cfc76f2418e-kube-api-access-kchwc\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.501781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.501777 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.501781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.501789 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz7wl\" (UniqueName: \"kubernetes.io/projected/cf31407f-9393-4e29-abd5-11332bf68e81-kube-api-access-zz7wl\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.502043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.501802 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.502043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.501814 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.502043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.501823 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf31407f-9393-4e29-abd5-11332bf68e81-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.502043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.501831 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.502043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.501838 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49b507f-a09f-4f74-8446-1cfc76f2418e-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.502043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.501846 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49b507f-a09f-4f74-8446-1cfc76f2418e-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.502043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.501854 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf31407f-9393-4e29-abd5-11332bf68e81-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:32.891154 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.891118 4883 generic.go:356] "Generic (PLEG): container finished" podID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerID="c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272" exitCode=0 Oct 30 18:40:32.891670 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.891339 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" Oct 30 18:40:32.892054 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.892020 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" event={"ID":"c49b507f-a09f-4f74-8446-1cfc76f2418e","Type":"ContainerDied","Data":"c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272"} Oct 30 18:40:32.892182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.892071 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm" event={"ID":"c49b507f-a09f-4f74-8446-1cfc76f2418e","Type":"ContainerDied","Data":"98a2360bda35a3c272073abbab4d519f3d7c7856732b6b8f00014399e5eaadaa"} Oct 30 18:40:32.892182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.892094 4883 scope.go:122] "RemoveContainer" containerID="c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272" Oct 30 18:40:32.893804 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.893779 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" event={"ID":"5ae7a355-7255-4fcf-b5dd-5bce0659f0df","Type":"ContainerDied","Data":"5302ad83da1e79ef9e5a420dc6cc4758b9215dc4949c28e5b030889c1c577b2a"} Oct 30 18:40:32.893909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.893828 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5302ad83da1e79ef9e5a420dc6cc4758b9215dc4949c28e5b030889c1c577b2a" Oct 30 18:40:32.893909 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.893886 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj" Oct 30 18:40:32.898540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.898518 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" Oct 30 18:40:32.898540 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.898535 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5" event={"ID":"211f0fa1-8896-4b7c-804b-9909d475c6e2","Type":"ContainerDied","Data":"2a4a09839eebc7328d237cbc8d4d68b8323efffd2a98b9cacfa1031fd508a291"} Oct 30 18:40:32.898714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.898572 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a4a09839eebc7328d237cbc8d4d68b8323efffd2a98b9cacfa1031fd508a291" Oct 30 18:40:32.900967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.900057 4883 generic.go:356] "Generic (PLEG): container finished" podID="cf31407f-9393-4e29-abd5-11332bf68e81" containerID="dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43" exitCode=0 Oct 30 18:40:32.900967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.900096 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" Oct 30 18:40:32.900967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.900158 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" event={"ID":"cf31407f-9393-4e29-abd5-11332bf68e81","Type":"ContainerDied","Data":"dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43"} Oct 30 18:40:32.900967 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.900185 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" event={"ID":"cf31407f-9393-4e29-abd5-11332bf68e81","Type":"ContainerDied","Data":"6a3f55ae163078f0925afe27aecddba58def7961aae3a7b9e35aa809c781aaad"} Oct 30 18:40:32.903820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.903801 4883 scope.go:122] "RemoveContainer" containerID="c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272" Oct 30 18:40:32.904161 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:40:32.904142 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272\": container with ID starting with c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272 not found: ID does not exist" containerID="c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272" Oct 30 18:40:32.904255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.904165 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272"} err="failed to get container status \"c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272\": rpc error: code = NotFound desc = could not find container \"c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272\": container with ID starting with c4a0ceaa3d0e5a951ca211f93409cd6f8eb02fc80d276b82ecf0629cd1aaa272 not found: ID does not exist" Oct 30 18:40:32.904255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.904184 4883 scope.go:122] "RemoveContainer" containerID="dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43" Oct 30 18:40:32.932626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.932607 4883 scope.go:122] "RemoveContainer" containerID="dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43" Oct 30 18:40:32.933005 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:40:32.932981 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43\": container with ID starting with dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43 not found: ID does not exist" containerID="dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43" Oct 30 18:40:32.933110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:32.933016 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43"} err="failed to get container status \"dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43\": rpc error: code = NotFound desc = could not find container \"dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43\": container with ID starting with dc45ce63166d1b07413c85628f212ca6ee04ab9d483912b7e5362167a4831b43 not found: ID does not exist" Oct 30 18:40:33.016155 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:33.016123 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm"] Oct 30 18:40:33.019156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:33.019131 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/distinct-2-w3-3cc458-kserve-784f5879b5-k68zm"] Oct 30 18:40:33.085958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:33.085784 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj"] Oct 30 18:40:33.093907 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:33.093876 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj"] Oct 30 18:40:33.195068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:33.194981 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" path="/var/lib/kubelet/pods/c49b507f-a09f-4f74-8446-1cfc76f2418e/volumes" Oct 30 18:40:33.195507 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:33.195487 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" path="/var/lib/kubelet/pods/cf31407f-9393-4e29-abd5-11332bf68e81/volumes" Oct 30 18:40:33.294801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:33.294748 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc458/simulator-w3-3cc458-kserve-b9598ddbb-4znlj" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:33.384483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:33.384436 4883 scope.go:122] "RemoveContainer" containerID="6b58cdc653507de7ad942051a7b808df10078c3752e1f251108d70572189db49" Oct 30 18:40:33.450342 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:33.450279 4883 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:33.450342 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:33.450326 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" podUID="5a60d9c1-369e-4818-8e6b-80e5bd537d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 18:40:34.239976 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.239584 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r"] Oct 30 18:40:34.240734 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.240711 4883 cpu_manager.go:414] "containerMap: removing container" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" Oct 30 18:40:34.240826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.240735 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" Oct 30 18:40:34.240826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.240772 4883 cpu_manager.go:414] "containerMap: removing container" podUID="211f0fa1-8896-4b7c-804b-9909d475c6e2" containerName="revoke-keys" Oct 30 18:40:34.240826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.240780 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211f0fa1-8896-4b7c-804b-9909d475c6e2" containerName="revoke-keys" Oct 30 18:40:34.240826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.240818 4883 cpu_manager.go:414] "containerMap: removing container" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" Oct 30 18:40:34.240826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.240826 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" Oct 30 18:40:34.241168 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.240899 4883 cpu_manager.go:414] "containerMap: removing container" podUID="5ae7a355-7255-4fcf-b5dd-5bce0659f0df" containerName="revoke-keys" Oct 30 18:40:34.241168 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.240906 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae7a355-7255-4fcf-b5dd-5bce0659f0df" containerName="revoke-keys" Oct 30 18:40:34.256959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.255750 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf31407f-9393-4e29-abd5-11332bf68e81" containerName="main" Oct 30 18:40:34.256959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.255806 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae7a355-7255-4fcf-b5dd-5bce0659f0df" containerName="revoke-keys" Oct 30 18:40:34.256959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.255820 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49b507f-a09f-4f74-8446-1cfc76f2418e" containerName="main" Oct 30 18:40:34.256959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.255841 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="211f0fa1-8896-4b7c-804b-9909d475c6e2" containerName="revoke-keys" Oct 30 18:40:34.299852 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.299810 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd"] Oct 30 18:40:34.300224 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.300200 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:34.313655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.308369 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:34.313655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.309158 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r"] Oct 30 18:40:34.313655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.313556 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd"] Oct 30 18:40:34.319526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.317462 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:40:34.319526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.318042 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:40:34.319526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.318123 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:40:34.319526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.318175 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:40:34.423157 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.423109 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns69v\" (UniqueName: \"kubernetes.io/projected/60578dc3-222d-4689-8103-e0782b2de3d1-kube-api-access-ns69v\") pod \"maas-api-revoke-sub-c26a381e-r4b4r\" (UID: \"60578dc3-222d-4689-8103-e0782b2de3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:34.423317 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.423184 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f3207b-hdnbd\" (UID: \"d394ecb7-4864-4387-88cb-a07bf6e39af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:34.423377 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.423316 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c26a381e-r4b4r\" (UID: \"60578dc3-222d-4689-8103-e0782b2de3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:34.423377 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.423368 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c26a381e-r4b4r\" (UID: \"60578dc3-222d-4689-8103-e0782b2de3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:34.423476 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.423457 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f3207b-hdnbd\" (UID: \"d394ecb7-4864-4387-88cb-a07bf6e39af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:34.423526 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.423512 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xnfr\" (UniqueName: \"kubernetes.io/projected/d394ecb7-4864-4387-88cb-a07bf6e39af1-kube-api-access-5xnfr\") pod \"maas-api-revoke-sub-83f3207b-hdnbd\" (UID: \"d394ecb7-4864-4387-88cb-a07bf6e39af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:34.450679 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.450646 4883 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:34.450832 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.450697 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" podUID="5a60d9c1-369e-4818-8e6b-80e5bd537d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 18:40:34.524894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.524806 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xnfr\" (UniqueName: \"kubernetes.io/projected/d394ecb7-4864-4387-88cb-a07bf6e39af1-kube-api-access-5xnfr\") pod \"maas-api-revoke-sub-83f3207b-hdnbd\" (UID: \"d394ecb7-4864-4387-88cb-a07bf6e39af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:34.524894 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.524849 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns69v\" (UniqueName: \"kubernetes.io/projected/60578dc3-222d-4689-8103-e0782b2de3d1-kube-api-access-ns69v\") pod \"maas-api-revoke-sub-c26a381e-r4b4r\" (UID: \"60578dc3-222d-4689-8103-e0782b2de3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:34.525116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.524904 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f3207b-hdnbd\" (UID: \"d394ecb7-4864-4387-88cb-a07bf6e39af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:34.525116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.525010 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c26a381e-r4b4r\" (UID: \"60578dc3-222d-4689-8103-e0782b2de3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:34.525116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.525061 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c26a381e-r4b4r\" (UID: \"60578dc3-222d-4689-8103-e0782b2de3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:34.525116 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.525109 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f3207b-hdnbd\" (UID: \"d394ecb7-4864-4387-88cb-a07bf6e39af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:34.525769 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.525739 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c26a381e-r4b4r\" (UID: \"60578dc3-222d-4689-8103-e0782b2de3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:34.525769 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.525755 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f3207b-hdnbd\" (UID: \"d394ecb7-4864-4387-88cb-a07bf6e39af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:34.527833 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.527807 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c26a381e-r4b4r\" (UID: \"60578dc3-222d-4689-8103-e0782b2de3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:34.527975 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.527926 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f3207b-hdnbd\" (UID: \"d394ecb7-4864-4387-88cb-a07bf6e39af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:34.553074 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.553038 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns69v\" (UniqueName: \"kubernetes.io/projected/60578dc3-222d-4689-8103-e0782b2de3d1-kube-api-access-ns69v\") pod \"maas-api-revoke-sub-c26a381e-r4b4r\" (UID: \"60578dc3-222d-4689-8103-e0782b2de3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:34.553348 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.553323 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xnfr\" (UniqueName: \"kubernetes.io/projected/d394ecb7-4864-4387-88cb-a07bf6e39af1-kube-api-access-5xnfr\") pod \"maas-api-revoke-sub-83f3207b-hdnbd\" (UID: \"d394ecb7-4864-4387-88cb-a07bf6e39af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:34.624637 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.624601 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:34.634725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.634691 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:34.956396 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.956365 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r"] Oct 30 18:40:34.958011 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:40:34.957976 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60578dc3_222d_4689_8103_e0782b2de3d1.slice/crio-778b6e3969f54ac93cb4e26a35075b71de0d21337b58fd784349e9f4350dad99 WatchSource:0}: Error finding container 778b6e3969f54ac93cb4e26a35075b71de0d21337b58fd784349e9f4350dad99: Status 404 returned error can't find the container with id 778b6e3969f54ac93cb4e26a35075b71de0d21337b58fd784349e9f4350dad99 Oct 30 18:40:34.992548 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:34.992516 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd"] Oct 30 18:40:34.995281 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:40:34.995253 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd394ecb7_4864_4387_88cb_a07bf6e39af1.slice/crio-eb149e81677fb5052ca635f30afeffcbd104e82a4dc266eb8cda8a904251e708 WatchSource:0}: Error finding container eb149e81677fb5052ca635f30afeffcbd104e82a4dc266eb8cda8a904251e708: Status 404 returned error can't find the container with id eb149e81677fb5052ca635f30afeffcbd104e82a4dc266eb8cda8a904251e708 Oct 30 18:40:35.110680 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:40:35.110645 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:40:35.450621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:35.450585 4883 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:35.451014 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:35.450638 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" podUID="5a60d9c1-369e-4818-8e6b-80e5bd537d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 18:40:35.922113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:35.922071 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" event={"ID":"d394ecb7-4864-4387-88cb-a07bf6e39af1","Type":"ContainerStarted","Data":"67288b89042056241d5d26fbe5c6b9ec0bc98564b9f8b2b3a97fd9dee75f48a6"} Oct 30 18:40:35.922317 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:35.922140 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" event={"ID":"d394ecb7-4864-4387-88cb-a07bf6e39af1","Type":"ContainerStarted","Data":"eb149e81677fb5052ca635f30afeffcbd104e82a4dc266eb8cda8a904251e708"} Oct 30 18:40:35.923828 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:35.923799 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" event={"ID":"60578dc3-222d-4689-8103-e0782b2de3d1","Type":"ContainerStarted","Data":"43d11bfb02068001263667c801619fd8bf2355767e1a9b59932a015d876f5c95"} Oct 30 18:40:35.923944 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:35.923849 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" event={"ID":"60578dc3-222d-4689-8103-e0782b2de3d1","Type":"ContainerStarted","Data":"778b6e3969f54ac93cb4e26a35075b71de0d21337b58fd784349e9f4350dad99"} Oct 30 18:40:35.990659 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:35.990033 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" podStartSLOduration=1.99000996 podStartE2EDuration="1.99000996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:35.975195332 +0000 UTC m=+2894.932199169" watchObservedRunningTime="2026-10-30 18:40:35.99000996 +0000 UTC m=+2894.947013796" Oct 30 18:40:36.021180 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:36.021126 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" podStartSLOduration=2.021109462 podStartE2EDuration="2.021109462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:36.018071217 +0000 UTC m=+2894.975075053" watchObservedRunningTime="2026-10-30 18:40:36.021109462 +0000 UTC m=+2894.978113297" Oct 30 18:40:36.450156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:36.450114 4883 patch_prober.go:29] interesting pod/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:40:36.450349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:36.450172 4883 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" podUID="5a60d9c1-369e-4818-8e6b-80e5bd537d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 18:40:36.931903 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:36.931867 4883 generic.go:356] "Generic (PLEG): container finished" podID="d394ecb7-4864-4387-88cb-a07bf6e39af1" containerID="67288b89042056241d5d26fbe5c6b9ec0bc98564b9f8b2b3a97fd9dee75f48a6" exitCode=0 Oct 30 18:40:36.932375 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:36.931965 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" event={"ID":"d394ecb7-4864-4387-88cb-a07bf6e39af1","Type":"ContainerDied","Data":"67288b89042056241d5d26fbe5c6b9ec0bc98564b9f8b2b3a97fd9dee75f48a6"} Oct 30 18:40:36.933647 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:36.933626 4883 generic.go:356] "Generic (PLEG): container finished" podID="60578dc3-222d-4689-8103-e0782b2de3d1" containerID="43d11bfb02068001263667c801619fd8bf2355767e1a9b59932a015d876f5c95" exitCode=0 Oct 30 18:40:36.933770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:36.933668 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" event={"ID":"60578dc3-222d-4689-8103-e0782b2de3d1","Type":"ContainerDied","Data":"43d11bfb02068001263667c801619fd8bf2355767e1a9b59932a015d876f5c95"} Oct 30 18:40:37.454516 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:37.454484 4883 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:37.454903 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:37.454878 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:37.455632 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:37.455612 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:40:38.124158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.124130 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:38.128138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.128115 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:38.177035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.173499 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60578dc3-222d-4689-8103-e0782b2de3d1-kube-api-access-ns69v\" (UniqueName: \"kubernetes.io/projected/60578dc3-222d-4689-8103-e0782b2de3d1-kube-api-access-ns69v\") pod \"60578dc3-222d-4689-8103-e0782b2de3d1\" (UID: \"60578dc3-222d-4689-8103-e0782b2de3d1\") " Oct 30 18:40:38.177035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.173633 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-xks-ca\") pod \"d394ecb7-4864-4387-88cb-a07bf6e39af1\" (UID: \"d394ecb7-4864-4387-88cb-a07bf6e39af1\") " Oct 30 18:40:38.177035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.173807 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-openshift-ca\") pod \"d394ecb7-4864-4387-88cb-a07bf6e39af1\" (UID: \"d394ecb7-4864-4387-88cb-a07bf6e39af1\") " Oct 30 18:40:38.177035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.173851 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-openshift-ca\") pod \"60578dc3-222d-4689-8103-e0782b2de3d1\" (UID: \"60578dc3-222d-4689-8103-e0782b2de3d1\") " Oct 30 18:40:38.177035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.173962 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-xks-ca\") pod \"60578dc3-222d-4689-8103-e0782b2de3d1\" (UID: \"60578dc3-222d-4689-8103-e0782b2de3d1\") " Oct 30 18:40:38.177035 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.174021 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d394ecb7-4864-4387-88cb-a07bf6e39af1-kube-api-access-5xnfr\" (UniqueName: \"kubernetes.io/projected/d394ecb7-4864-4387-88cb-a07bf6e39af1-kube-api-access-5xnfr\") pod \"d394ecb7-4864-4387-88cb-a07bf6e39af1\" (UID: \"d394ecb7-4864-4387-88cb-a07bf6e39af1\") " Oct 30 18:40:38.197165 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.184212 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-openshift-ca" pod "60578dc3-222d-4689-8103-e0782b2de3d1" (UID: "60578dc3-222d-4689-8103-e0782b2de3d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:38.197386 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.197191 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-openshift-ca" pod "d394ecb7-4864-4387-88cb-a07bf6e39af1" (UID: "d394ecb7-4864-4387-88cb-a07bf6e39af1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:38.211255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.209102 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-xks-ca" pod "d394ecb7-4864-4387-88cb-a07bf6e39af1" (UID: "d394ecb7-4864-4387-88cb-a07bf6e39af1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:38.211887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.211787 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60578dc3-222d-4689-8103-e0782b2de3d1-kube-api-access-ns69v" pod "60578dc3-222d-4689-8103-e0782b2de3d1" (UID: "60578dc3-222d-4689-8103-e0782b2de3d1"). InnerVolumeSpecName "kube-api-access-ns69v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:38.211887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.211797 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-xks-ca" pod "60578dc3-222d-4689-8103-e0782b2de3d1" (UID: "60578dc3-222d-4689-8103-e0782b2de3d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:38.211887 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.211838 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d394ecb7-4864-4387-88cb-a07bf6e39af1-kube-api-access-5xnfr" pod "d394ecb7-4864-4387-88cb-a07bf6e39af1" (UID: "d394ecb7-4864-4387-88cb-a07bf6e39af1"). InnerVolumeSpecName "kube-api-access-5xnfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:38.280432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.280387 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:38.280432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.280422 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:38.280432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.280432 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60578dc3-222d-4689-8103-e0782b2de3d1-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:38.280432 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.280441 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xnfr\" (UniqueName: \"kubernetes.io/projected/d394ecb7-4864-4387-88cb-a07bf6e39af1-kube-api-access-5xnfr\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:38.280756 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.280450 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns69v\" (UniqueName: \"kubernetes.io/projected/60578dc3-222d-4689-8103-e0782b2de3d1-kube-api-access-ns69v\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:38.280756 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.280461 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d394ecb7-4864-4387-88cb-a07bf6e39af1-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:38.680881 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.680811 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" containerID="cri-o://4a939b0c861da5c50250ffb4e1e2fb0357edc2289451b10c531b06542dcc6579" gracePeriod=45 Oct 30 18:40:38.704437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.704385 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" containerID="cri-o://c5bba1252b3e325682332247360375920c2575d8e9136843ec4d3e9346f4f51b" gracePeriod=45 Oct 30 18:40:38.708094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.708059 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" containerID="cri-o://6ae38f383fed3d0da9688e8e00dafe14ad6625f7ed23e538885672d4ce1808f7" gracePeriod=45 Oct 30 18:40:38.855974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.853921 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": dial tcp 10.132.0.99:8000: connect: connection refused" Oct 30 18:40:38.952129 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.952040 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" Oct 30 18:40:38.952848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.952551 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r" event={"ID":"60578dc3-222d-4689-8103-e0782b2de3d1","Type":"ContainerDied","Data":"778b6e3969f54ac93cb4e26a35075b71de0d21337b58fd784349e9f4350dad99"} Oct 30 18:40:38.952848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.952606 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="778b6e3969f54ac93cb4e26a35075b71de0d21337b58fd784349e9f4350dad99" Oct 30 18:40:38.954439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.954411 4883 generic.go:356] "Generic (PLEG): container finished" podID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerID="c5bba1252b3e325682332247360375920c2575d8e9136843ec4d3e9346f4f51b" exitCode=0 Oct 30 18:40:38.954549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.954501 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" event={"ID":"39fe754a-78b0-4d4e-8275-7ea2dd247508","Type":"ContainerDied","Data":"c5bba1252b3e325682332247360375920c2575d8e9136843ec4d3e9346f4f51b"} Oct 30 18:40:38.957161 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.957133 4883 generic.go:356] "Generic (PLEG): container finished" podID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerID="4a939b0c861da5c50250ffb4e1e2fb0357edc2289451b10c531b06542dcc6579" exitCode=0 Oct 30 18:40:38.957255 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.957214 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" event={"ID":"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed","Type":"ContainerDied","Data":"4a939b0c861da5c50250ffb4e1e2fb0357edc2289451b10c531b06542dcc6579"} Oct 30 18:40:38.959834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.959155 4883 generic.go:356] "Generic (PLEG): container finished" podID="eff20d9a-c474-4b76-809d-add74ef702af" containerID="6ae38f383fed3d0da9688e8e00dafe14ad6625f7ed23e538885672d4ce1808f7" exitCode=0 Oct 30 18:40:38.959834 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.959249 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" event={"ID":"eff20d9a-c474-4b76-809d-add74ef702af","Type":"ContainerDied","Data":"6ae38f383fed3d0da9688e8e00dafe14ad6625f7ed23e538885672d4ce1808f7"} Oct 30 18:40:38.961132 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.961113 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" Oct 30 18:40:38.961223 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.961201 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd" event={"ID":"d394ecb7-4864-4387-88cb-a07bf6e39af1","Type":"ContainerDied","Data":"eb149e81677fb5052ca635f30afeffcbd104e82a4dc266eb8cda8a904251e708"} Oct 30 18:40:38.961278 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.961225 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb149e81677fb5052ca635f30afeffcbd104e82a4dc266eb8cda8a904251e708" Oct 30 18:40:38.971750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.971722 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:40:38.991263 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.991230 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-kube-api-access-sdp2z\" (UniqueName: \"kubernetes.io/projected/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-kube-api-access-sdp2z\") pod \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " Oct 30 18:40:38.991439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.991281 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-model-cache\") pod \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " Oct 30 18:40:38.991439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.991306 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tmp-dir\") pod \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " Oct 30 18:40:38.991439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.991352 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-dshm\") pod \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " Oct 30 18:40:38.991919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.991502 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tls-certs\") pod \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " Oct 30 18:40:38.991919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.991535 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-home\") pod \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\" (UID: \"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed\") " Oct 30 18:40:38.991919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.991606 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tmp-dir" pod "9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" (UID: "9f2e9af9-7137-4b2e-a0ea-beea94c4cbed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:38.991919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.991858 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-home" pod "9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" (UID: "9f2e9af9-7137-4b2e-a0ea-beea94c4cbed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:38.991919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.991896 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-model-cache" pod "9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" (UID: "9f2e9af9-7137-4b2e-a0ea-beea94c4cbed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:38.992251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.992127 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:38.992251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.992147 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:38.992251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.992160 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:38.997285 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.997241 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-dshm" pod "9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" (UID: "9f2e9af9-7137-4b2e-a0ea-beea94c4cbed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:38.997737 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.997711 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-kube-api-access-sdp2z" pod "9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" (UID: "9f2e9af9-7137-4b2e-a0ea-beea94c4cbed"). InnerVolumeSpecName "kube-api-access-sdp2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:38.998076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:38.998054 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tls-certs" pod "9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" (UID: "9f2e9af9-7137-4b2e-a0ea-beea94c4cbed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:39.009499 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.009472 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:40:39.040998 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.036000 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:40:39.093249 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.093219 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-tmp-dir\") pod \"39fe754a-78b0-4d4e-8275-7ea2dd247508\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " Oct 30 18:40:39.093409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.093269 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-model-cache\") pod \"39fe754a-78b0-4d4e-8275-7ea2dd247508\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " Oct 30 18:40:39.093409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.093317 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-dshm\") pod \"39fe754a-78b0-4d4e-8275-7ea2dd247508\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " Oct 30 18:40:39.093409 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.093350 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-dshm\") pod \"eff20d9a-c474-4b76-809d-add74ef702af\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " Oct 30 18:40:39.093912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.093419 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eff20d9a-c474-4b76-809d-add74ef702af-tls-certs\" (UniqueName: \"kubernetes.io/secret/eff20d9a-c474-4b76-809d-add74ef702af-tls-certs\") pod \"eff20d9a-c474-4b76-809d-add74ef702af\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " Oct 30 18:40:39.093912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.093471 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39fe754a-78b0-4d4e-8275-7ea2dd247508-kube-api-access-qbz49\" (UniqueName: \"kubernetes.io/projected/39fe754a-78b0-4d4e-8275-7ea2dd247508-kube-api-access-qbz49\") pod \"39fe754a-78b0-4d4e-8275-7ea2dd247508\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " Oct 30 18:40:39.093912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.093478 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-tmp-dir" pod "39fe754a-78b0-4d4e-8275-7ea2dd247508" (UID: "39fe754a-78b0-4d4e-8275-7ea2dd247508"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:39.093912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.093693 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-home" pod "eff20d9a-c474-4b76-809d-add74ef702af" (UID: "eff20d9a-c474-4b76-809d-add74ef702af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:39.093912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.093562 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-model-cache" pod "39fe754a-78b0-4d4e-8275-7ea2dd247508" (UID: "39fe754a-78b0-4d4e-8275-7ea2dd247508"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:39.093912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.093505 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-home\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-home\") pod \"eff20d9a-c474-4b76-809d-add74ef702af\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " Oct 30 18:40:39.093912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.093859 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-model-cache\") pod \"eff20d9a-c474-4b76-809d-add74ef702af\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " Oct 30 18:40:39.093912 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.093898 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff20d9a-c474-4b76-809d-add74ef702af-kube-api-access-5vmvb\" (UniqueName: \"kubernetes.io/projected/eff20d9a-c474-4b76-809d-add74ef702af-kube-api-access-5vmvb\") pod \"eff20d9a-c474-4b76-809d-add74ef702af\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " Oct 30 18:40:39.094387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.094009 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-tmp-dir\") pod \"eff20d9a-c474-4b76-809d-add74ef702af\" (UID: \"eff20d9a-c474-4b76-809d-add74ef702af\") " Oct 30 18:40:39.094387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.094045 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39fe754a-78b0-4d4e-8275-7ea2dd247508-tls-certs\" (UniqueName: \"kubernetes.io/secret/39fe754a-78b0-4d4e-8275-7ea2dd247508-tls-certs\") pod \"39fe754a-78b0-4d4e-8275-7ea2dd247508\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " Oct 30 18:40:39.094387 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.094086 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-home\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-home\") pod \"39fe754a-78b0-4d4e-8275-7ea2dd247508\" (UID: \"39fe754a-78b0-4d4e-8275-7ea2dd247508\") " Oct 30 18:40:39.094825 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.094780 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-model-cache" pod "eff20d9a-c474-4b76-809d-add74ef702af" (UID: "eff20d9a-c474-4b76-809d-add74ef702af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:39.094825 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.094792 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-tmp-dir" pod "eff20d9a-c474-4b76-809d-add74ef702af" (UID: "eff20d9a-c474-4b76-809d-add74ef702af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:39.095152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.095090 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-home" pod "39fe754a-78b0-4d4e-8275-7ea2dd247508" (UID: "39fe754a-78b0-4d4e-8275-7ea2dd247508"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:39.095664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.095529 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.095664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.095557 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.095664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.095570 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.095664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.095583 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.095664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.095596 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.095664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.095611 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdp2z\" (UniqueName: \"kubernetes.io/projected/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-kube-api-access-sdp2z\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.095664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.095624 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.095664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.095636 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.095664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.095646 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.096561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.096496 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-dshm" pod "eff20d9a-c474-4b76-809d-add74ef702af" (UID: "eff20d9a-c474-4b76-809d-add74ef702af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:39.097143 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.097116 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff20d9a-c474-4b76-809d-add74ef702af-kube-api-access-5vmvb" pod "eff20d9a-c474-4b76-809d-add74ef702af" (UID: "eff20d9a-c474-4b76-809d-add74ef702af"). InnerVolumeSpecName "kube-api-access-5vmvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:39.097684 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.097658 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff20d9a-c474-4b76-809d-add74ef702af-tls-certs" pod "eff20d9a-c474-4b76-809d-add74ef702af" (UID: "eff20d9a-c474-4b76-809d-add74ef702af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:39.098375 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.098355 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39fe754a-78b0-4d4e-8275-7ea2dd247508-tls-certs" pod "39fe754a-78b0-4d4e-8275-7ea2dd247508" (UID: "39fe754a-78b0-4d4e-8275-7ea2dd247508"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:39.098529 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.098509 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-dshm" pod "39fe754a-78b0-4d4e-8275-7ea2dd247508" (UID: "39fe754a-78b0-4d4e-8275-7ea2dd247508"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:39.099122 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.099102 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39fe754a-78b0-4d4e-8275-7ea2dd247508-kube-api-access-qbz49" pod "39fe754a-78b0-4d4e-8275-7ea2dd247508" (UID: "39fe754a-78b0-4d4e-8275-7ea2dd247508"). InnerVolumeSpecName "kube-api-access-qbz49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:39.196490 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.196463 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39fe754a-78b0-4d4e-8275-7ea2dd247508-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.196820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.196494 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff20d9a-c474-4b76-809d-add74ef702af-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.196820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.196508 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff20d9a-c474-4b76-809d-add74ef702af-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.196820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.196524 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbz49\" (UniqueName: \"kubernetes.io/projected/39fe754a-78b0-4d4e-8275-7ea2dd247508-kube-api-access-qbz49\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.196820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.196539 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vmvb\" (UniqueName: \"kubernetes.io/projected/eff20d9a-c474-4b76-809d-add74ef702af-kube-api-access-5vmvb\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.196820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.196556 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39fe754a-78b0-4d4e-8275-7ea2dd247508-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:39.969217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.969188 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" Oct 30 18:40:39.969428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.969222 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns" event={"ID":"eff20d9a-c474-4b76-809d-add74ef702af","Type":"ContainerDied","Data":"0a8ed2bf1fd651cb7449144a1760b708e4330dbf8ed0fc5332de373b8fa13149"} Oct 30 18:40:39.969428 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.969279 4883 scope.go:122] "RemoveContainer" containerID="6ae38f383fed3d0da9688e8e00dafe14ad6625f7ed23e538885672d4ce1808f7" Oct 30 18:40:39.970820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.970796 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" Oct 30 18:40:39.970820 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.970813 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8" event={"ID":"39fe754a-78b0-4d4e-8275-7ea2dd247508","Type":"ContainerDied","Data":"1992f9ef2cdf6fd34ac4e3d0770f6717b8a66c70362e0e2080df2e7cd4ec4cdc"} Oct 30 18:40:39.972350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.972324 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" event={"ID":"9f2e9af9-7137-4b2e-a0ea-beea94c4cbed","Type":"ContainerDied","Data":"e0c923d3571e98bfcf8ec6c9dc488a200b10cd52c5514e127ff9e899c3eda013"} Oct 30 18:40:39.972350 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.972342 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr" Oct 30 18:40:39.980128 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.980110 4883 scope.go:122] "RemoveContainer" containerID="c5bba1252b3e325682332247360375920c2575d8e9136843ec4d3e9346f4f51b" Oct 30 18:40:39.988897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:39.988867 4883 scope.go:122] "RemoveContainer" containerID="4a939b0c861da5c50250ffb4e1e2fb0357edc2289451b10c531b06542dcc6579" Oct 30 18:40:40.025891 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:40.025852 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns"] Oct 30 18:40:40.038564 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:40.038533 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/simulator-w4-419d32-kserve-8dc97d64b-r6zns"] Oct 30 18:40:40.066216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:40.066180 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr"] Oct 30 18:40:40.070625 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:40.070601 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/distinct-2-w4-419d32-kserve-865477b849-tzvwr"] Oct 30 18:40:40.107636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:40.107607 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8"] Oct 30 18:40:40.111610 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:40.111584 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-419d32/distinct-w4-419d32-kserve-5768cb7fd-d48j8"] Oct 30 18:40:41.180437 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:41.180399 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" path="/var/lib/kubelet/pods/39fe754a-78b0-4d4e-8275-7ea2dd247508/volumes" Oct 30 18:40:41.181131 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:41.181109 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" path="/var/lib/kubelet/pods/9f2e9af9-7137-4b2e-a0ea-beea94c4cbed/volumes" Oct 30 18:40:41.181675 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:41.181657 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff20d9a-c474-4b76-809d-add74ef702af" path="/var/lib/kubelet/pods/eff20d9a-c474-4b76-809d-add74ef702af/volumes" Oct 30 18:40:41.934228 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:41.934188 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4"] Oct 30 18:40:41.942851 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:41.942817 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3cc458-7r2g4"] Oct 30 18:40:43.179750 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:43.179715 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e86dbd2-1be5-44ba-9a26-eef77565a328" path="/var/lib/kubelet/pods/4e86dbd2-1be5-44ba-9a26-eef77565a328/volumes" Oct 30 18:40:47.677346 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:47.677284 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw"] Oct 30 18:40:47.689847 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:47.689800 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-419d32-fszxw"] Oct 30 18:40:49.178036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:49.177999 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00efc6b-3f0f-4e79-8a99-21be3e64a24d" path="/var/lib/kubelet/pods/a00efc6b-3f0f-4e79-8a99-21be3e64a24d/volumes" Oct 30 18:40:50.238290 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:50.238251 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz"] Oct 30 18:40:50.239156 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:50.239099 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" containerID="cri-o://4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a" gracePeriod=30 Oct 30 18:40:50.250297 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:50.250260 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw"] Oct 30 18:40:50.250595 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:50.250546 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" containerID="cri-o://00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0" gracePeriod=30 Oct 30 18:40:50.254844 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:50.254575 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm"] Oct 30 18:40:50.255259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:50.255227 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" containerID="cri-o://c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379" gracePeriod=30 Oct 30 18:40:55.610439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.610076 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.755657 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-envoy\") pod \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.755753 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-kube-api-access-8f448\" (UniqueName: \"kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-kube-api-access-8f448\") pod \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.755785 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-certs\") pod \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.755874 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-podinfo\") pod \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.755913 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-credential-socket\") pod \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.755960 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-data\") pod \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.756037 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-socket\") pod \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.756097 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istiod-ca-cert\") pod \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.756139 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-token\" (UniqueName: \"kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-token\") pod \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\" (UID: \"40b5a129-7409-43f5-8ffc-5c6b7738cdb4\") " Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.769005 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-data" pod "40b5a129-7409-43f5-8ffc-5c6b7738cdb4" (UID: "40b5a129-7409-43f5-8ffc-5c6b7738cdb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.769139 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-socket" pod "40b5a129-7409-43f5-8ffc-5c6b7738cdb4" (UID: "40b5a129-7409-43f5-8ffc-5c6b7738cdb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.769329 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istiod-ca-cert" pod "40b5a129-7409-43f5-8ffc-5c6b7738cdb4" (UID: "40b5a129-7409-43f5-8ffc-5c6b7738cdb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.769580 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-certs" pod "40b5a129-7409-43f5-8ffc-5c6b7738cdb4" (UID: "40b5a129-7409-43f5-8ffc-5c6b7738cdb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.787604 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.779837 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-credential-socket" pod "40b5a129-7409-43f5-8ffc-5c6b7738cdb4" (UID: "40b5a129-7409-43f5-8ffc-5c6b7738cdb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.804250 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.804146 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-podinfo" pod "40b5a129-7409-43f5-8ffc-5c6b7738cdb4" (UID: "40b5a129-7409-43f5-8ffc-5c6b7738cdb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:55.854293 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.816134 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-token" pod "40b5a129-7409-43f5-8ffc-5c6b7738cdb4" (UID: "40b5a129-7409-43f5-8ffc-5c6b7738cdb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:55.868046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.860046 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.868046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.860094 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.868046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.860111 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.868046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.860127 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.868046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.860142 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.868046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.860157 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.868046 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.860172 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.904297 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.904232 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-envoy" pod "40b5a129-7409-43f5-8ffc-5c6b7738cdb4" (UID: "40b5a129-7409-43f5-8ffc-5c6b7738cdb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.909010 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.908948 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-kube-api-access-8f448" pod "40b5a129-7409-43f5-8ffc-5c6b7738cdb4" (UID: "40b5a129-7409-43f5-8ffc-5c6b7738cdb4"). InnerVolumeSpecName "kube-api-access-8f448". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:55.929960 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.926726 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:40:55.932030 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.931831 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.963530 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-socket\") pod \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.963591 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-token\" (UniqueName: \"kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-token\") pod \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.963625 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-kube-api-access-4wchh\" (UniqueName: \"kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-kube-api-access-4wchh\") pod \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.963656 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-kube-api-access-wdchz\" (UniqueName: \"kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-kube-api-access-wdchz\") pod \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.963695 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-credential-socket\") pod \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.963746 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-token\" (UniqueName: \"kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-token\") pod \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.963771 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-data\") pod \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.963795 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-certs\") pod \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.963828 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-data\") pod \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.963858 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-envoy\") pod \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.963883 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-envoy\") pod \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.963981 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-podinfo\") pod \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.964026 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-credential-socket\") pod \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.964098 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-certs\") pod \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.964133 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-socket\") pod \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.964197 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-podinfo\") pod \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.964269 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istiod-ca-cert\") pod \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\" (UID: \"7edf8fc2-6fba-401b-8f7e-6b73830e1745\") " Oct 30 18:40:55.964958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.964310 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istiod-ca-cert\") pod \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\" (UID: \"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0\") " Oct 30 18:40:55.975443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.974743 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-data" pod "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" (UID: "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.975443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.975004 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-socket" pod "7edf8fc2-6fba-401b-8f7e-6b73830e1745" (UID: "7edf8fc2-6fba-401b-8f7e-6b73830e1745"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.975443 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.975189 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-credential-socket" pod "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" (UID: "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.976109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.975788 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istiod-ca-cert" pod "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" (UID: "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:55.976109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.976000 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.976109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.976019 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.976109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.976032 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.976109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.976044 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.976109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.976058 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f448\" (UniqueName: \"kubernetes.io/projected/40b5a129-7409-43f5-8ffc-5c6b7738cdb4-kube-api-access-8f448\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.976109 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.976071 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:55.976581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.976233 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-certs" pod "7edf8fc2-6fba-401b-8f7e-6b73830e1745" (UID: "7edf8fc2-6fba-401b-8f7e-6b73830e1745"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.976581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.976464 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-data" pod "7edf8fc2-6fba-401b-8f7e-6b73830e1745" (UID: "7edf8fc2-6fba-401b-8f7e-6b73830e1745"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.977184 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.976898 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-credential-socket" pod "7edf8fc2-6fba-401b-8f7e-6b73830e1745" (UID: "7edf8fc2-6fba-401b-8f7e-6b73830e1745"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.983809 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.983389 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istiod-ca-cert" pod "7edf8fc2-6fba-401b-8f7e-6b73830e1745" (UID: "7edf8fc2-6fba-401b-8f7e-6b73830e1745"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:55.987723 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.987662 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-socket" pod "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" (UID: "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:55.999862 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:55.999807 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-kube-api-access-wdchz" pod "7edf8fc2-6fba-401b-8f7e-6b73830e1745" (UID: "7edf8fc2-6fba-401b-8f7e-6b73830e1745"). InnerVolumeSpecName "kube-api-access-wdchz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:56.001919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.001882 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-podinfo" pod "7edf8fc2-6fba-401b-8f7e-6b73830e1745" (UID: "7edf8fc2-6fba-401b-8f7e-6b73830e1745"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:56.002254 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.002221 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-kube-api-access-4wchh" pod "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" (UID: "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0"). InnerVolumeSpecName "kube-api-access-4wchh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:56.003415 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.003378 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-token" pod "7edf8fc2-6fba-401b-8f7e-6b73830e1745" (UID: "7edf8fc2-6fba-401b-8f7e-6b73830e1745"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:56.003701 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.003666 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-token" pod "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" (UID: "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:56.004083 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.004050 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-podinfo" pod "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" (UID: "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:56.004298 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.004268 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-envoy" pod "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" (UID: "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.036009 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.015099 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-certs" pod "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" (UID: "349475cf-93a4-49d8-8f2f-61bb0b5b7bc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.089905 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.089958 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wchh\" (UniqueName: \"kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-kube-api-access-4wchh\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.089974 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdchz\" (UniqueName: \"kubernetes.io/projected/7edf8fc2-6fba-401b-8f7e-6b73830e1745-kube-api-access-wdchz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.089987 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.089999 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.090009 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.090019 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.090030 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.090040 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.090052 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.090063 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.090074 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.125158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.090087 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.312427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.312347 4883 generic.go:356] "Generic (PLEG): container finished" podID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerID="c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379" exitCode=0 Oct 30 18:40:56.316115 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.316012 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" event={"ID":"40b5a129-7409-43f5-8ffc-5c6b7738cdb4","Type":"ContainerDied","Data":"c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379"} Oct 30 18:40:56.406057 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.324214 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" Oct 30 18:40:56.406057 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.342568 4883 scope.go:122] "RemoveContainer" containerID="c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379" Oct 30 18:40:56.406057 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.343001 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm" event={"ID":"40b5a129-7409-43f5-8ffc-5c6b7738cdb4","Type":"ContainerDied","Data":"88572bdeb772a472aa0bee6906aadfb956914d2cb2bf22ee51747161b174946a"} Oct 30 18:40:56.458974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.443712 4883 generic.go:356] "Generic (PLEG): container finished" podID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerID="00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0" exitCode=0 Oct 30 18:40:56.458974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.443824 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" event={"ID":"7edf8fc2-6fba-401b-8f7e-6b73830e1745","Type":"ContainerDied","Data":"00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0"} Oct 30 18:40:56.458974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.443879 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" event={"ID":"7edf8fc2-6fba-401b-8f7e-6b73830e1745","Type":"ContainerDied","Data":"7009f008f11ae33aee8b70e5318f1a9309962cd33ea662b11f2a754cfcf4554e"} Oct 30 18:40:56.458974 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.443994 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw" Oct 30 18:40:56.519045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.459608 4883 generic.go:356] "Generic (PLEG): container finished" podID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerID="4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a" exitCode=0 Oct 30 18:40:56.519045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.459733 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" event={"ID":"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0","Type":"ContainerDied","Data":"4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a"} Oct 30 18:40:56.519045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.459782 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" event={"ID":"349475cf-93a4-49d8-8f2f-61bb0b5b7bc0","Type":"ContainerDied","Data":"bae0d150331485f1a338d041fd9596c96a8814bd44fee2325e16a5296a972f13"} Oct 30 18:40:56.519045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.459866 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz" Oct 30 18:40:56.520763 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.519484 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc"] Oct 30 18:40:56.520763 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.519787 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" containerID="cri-o://3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7" gracePeriod=30 Oct 30 18:40:56.573874 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.573833 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs"] Oct 30 18:40:56.649202 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.649160 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f"] Oct 30 18:40:56.650882 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.650649 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" containerID="cri-o://105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2" gracePeriod=30 Oct 30 18:40:56.650882 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.650129 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" containerID="cri-o://5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46" gracePeriod=30 Oct 30 18:40:56.855110 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.855013 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm"] Oct 30 18:40:56.857253 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.857220 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-envoy" pod "7edf8fc2-6fba-401b-8f7e-6b73830e1745" (UID: "7edf8fc2-6fba-401b-8f7e-6b73830e1745"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:56.910019 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.909984 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-7dtrm"] Oct 30 18:40:56.922959 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.910928 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edf8fc2-6fba-401b-8f7e-6b73830e1745-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:40:56.985472 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:56.985437 4883 scope.go:122] "RemoveContainer" containerID="c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379" Oct 30 18:40:57.011744 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:40:57.011705 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379\": container with ID starting with c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379 not found: ID does not exist" containerID="c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379" Oct 30 18:40:57.012150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.012097 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379"} err="failed to get container status \"c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379\": rpc error: code = NotFound desc = could not find container \"c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379\": container with ID starting with c8356404e121b38541dcdd09ce1a50e995921c56a839631d3821f4017e0a2379 not found: ID does not exist" Oct 30 18:40:57.012314 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.012297 4883 scope.go:122] "RemoveContainer" containerID="00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0" Oct 30 18:40:57.039087 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.039065 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz"] Oct 30 18:40:57.063172 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.063089 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-lllmz"] Oct 30 18:40:57.139718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.139384 4883 scope.go:122] "RemoveContainer" containerID="00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0" Oct 30 18:40:57.140864 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:40:57.140456 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0\": container with ID starting with 00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0 not found: ID does not exist" containerID="00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0" Oct 30 18:40:57.140864 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.140510 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0"} err="failed to get container status \"00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0\": rpc error: code = NotFound desc = could not find container \"00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0\": container with ID starting with 00a27ce26efc685f2f76e5cb4feb718f301123c8469b294159162d20f43012c0 not found: ID does not exist" Oct 30 18:40:57.140864 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.140535 4883 scope.go:122] "RemoveContainer" containerID="4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a" Oct 30 18:40:57.156627 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.156332 4883 scope.go:122] "RemoveContainer" containerID="4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a" Oct 30 18:40:57.156966 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:40:57.156838 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a\": container with ID starting with 4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a not found: ID does not exist" containerID="4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a" Oct 30 18:40:57.156966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.156878 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a"} err="failed to get container status \"4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a\": rpc error: code = NotFound desc = could not find container \"4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a\": container with ID starting with 4e1ada92dee1dd3afe2396f7a6b25c9ce38bd5f725c3b8dc150b6050a2c9ff5a not found: ID does not exist" Oct 30 18:40:57.186422 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.186387 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" path="/var/lib/kubelet/pods/349475cf-93a4-49d8-8f2f-61bb0b5b7bc0/volumes" Oct 30 18:40:57.197988 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.197920 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" path="/var/lib/kubelet/pods/40b5a129-7409-43f5-8ffc-5c6b7738cdb4/volumes" Oct 30 18:40:57.198587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.198572 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw"] Oct 30 18:40:57.270267 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:57.270226 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc458-openshift-default-75dc4c6579-hlsvw"] Oct 30 18:40:59.177789 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:40:59.177756 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" path="/var/lib/kubelet/pods/7edf8fc2-6fba-401b-8f7e-6b73830e1745/volumes" Oct 30 18:41:00.206138 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.206097 4883 patch_prober.go:29] interesting pod/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:41:00.206564 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.206180 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:41:00.390270 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.390225 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5"] Oct 30 18:41:00.390913 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.390890 4883 cpu_manager.go:414] "containerMap: removing container" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" Oct 30 18:41:00.390913 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.390910 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.390925 4883 cpu_manager.go:414] "containerMap: removing container" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.390945 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.390958 4883 cpu_manager.go:414] "containerMap: removing container" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.390965 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.390981 4883 cpu_manager.go:414] "containerMap: removing container" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.390986 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391004 4883 cpu_manager.go:414] "containerMap: removing container" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391010 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391022 4883 cpu_manager.go:414] "containerMap: removing container" podUID="60578dc3-222d-4689-8103-e0782b2de3d1" containerName="revoke-keys" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391027 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60578dc3-222d-4689-8103-e0782b2de3d1" containerName="revoke-keys" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391036 4883 cpu_manager.go:414] "containerMap: removing container" podUID="d394ecb7-4864-4387-88cb-a07bf6e39af1" containerName="revoke-keys" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391040 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d394ecb7-4864-4387-88cb-a07bf6e39af1" containerName="revoke-keys" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391052 4883 cpu_manager.go:414] "containerMap: removing container" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" Oct 30 18:41:00.391090 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391057 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" Oct 30 18:41:00.391527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391124 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b5a129-7409-43f5-8ffc-5c6b7738cdb4" containerName="istio-proxy" Oct 30 18:41:00.391527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391133 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="349475cf-93a4-49d8-8f2f-61bb0b5b7bc0" containerName="istio-proxy" Oct 30 18:41:00.391527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391141 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2e9af9-7137-4b2e-a0ea-beea94c4cbed" containerName="main" Oct 30 18:41:00.391527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391147 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="39fe754a-78b0-4d4e-8275-7ea2dd247508" containerName="main" Oct 30 18:41:00.391527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391156 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff20d9a-c474-4b76-809d-add74ef702af" containerName="main" Oct 30 18:41:00.391527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391163 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="60578dc3-222d-4689-8103-e0782b2de3d1" containerName="revoke-keys" Oct 30 18:41:00.391527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391169 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="d394ecb7-4864-4387-88cb-a07bf6e39af1" containerName="revoke-keys" Oct 30 18:41:00.391527 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.391177 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="7edf8fc2-6fba-401b-8f7e-6b73830e1745" containerName="istio-proxy" Oct 30 18:41:00.396700 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.396672 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:00.412664 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.412636 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:41:00.412803 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.412696 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:41:00.412803 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.412695 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:41:00.412803 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.412778 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:41:00.416128 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.416101 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5"] Oct 30 18:41:00.425510 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.425476 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5\" (UID: \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:00.425655 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.425526 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4klst\" (UniqueName: \"kubernetes.io/projected/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-kube-api-access-4klst\") pod \"maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5\" (UID: \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:00.425726 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.425688 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5\" (UID: \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:00.526910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.526820 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5\" (UID: \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:00.527112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.526984 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5\" (UID: \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:00.527112 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.527005 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4klst\" (UniqueName: \"kubernetes.io/projected/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-kube-api-access-4klst\") pod \"maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5\" (UID: \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:00.527631 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.527608 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5\" (UID: \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:00.529345 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.529322 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5\" (UID: \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:00.551885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.551853 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4klst\" (UniqueName: \"kubernetes.io/projected/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-kube-api-access-4klst\") pod \"maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5\" (UID: \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:00.713026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:00.712997 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:01.051472 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:01.051419 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5"] Oct 30 18:41:01.054674 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:41:01.054631 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb7a3a9_2369_4dbb_b045_a75ebe87fdb7.slice/crio-5e8f2833cb8cff6f632fad2478c9566933e54ff6ad6aad11ca2bc722206a5bff WatchSource:0}: Error finding container 5e8f2833cb8cff6f632fad2478c9566933e54ff6ad6aad11ca2bc722206a5bff: Status 404 returned error can't find the container with id 5e8f2833cb8cff6f632fad2478c9566933e54ff6ad6aad11ca2bc722206a5bff Oct 30 18:41:01.507096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:01.507055 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" event={"ID":"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7","Type":"ContainerStarted","Data":"be206ca6bb7dfac284e81869b7a9b206816729ae22e4f9a9388e0079a0b8a609"} Oct 30 18:41:01.507580 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:01.507115 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" event={"ID":"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7","Type":"ContainerStarted","Data":"5e8f2833cb8cff6f632fad2478c9566933e54ff6ad6aad11ca2bc722206a5bff"} Oct 30 18:41:01.556516 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:01.556464 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" podStartSLOduration=1.556449153 podStartE2EDuration="1.556449153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:01.553681366 +0000 UTC m=+2920.510685202" watchObservedRunningTime="2026-10-30 18:41:01.556449153 +0000 UTC m=+2920.513452985" Oct 30 18:41:02.048121 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.048088 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:41:02.051642 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.051617 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:41:02.142742 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.142709 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-data\") pod \"cde015c2-b427-435f-87bf-935a6056e8b9\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " Oct 30 18:41:02.142928 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.142754 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-certs\") pod \"cde015c2-b427-435f-87bf-935a6056e8b9\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " Oct 30 18:41:02.142928 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.142790 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-istio-token\" (UniqueName: \"kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-istio-token\") pod \"0789a37e-7327-47e9-9bc1-247533139ae4\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " Oct 30 18:41:02.142928 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.142818 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde015c2-b427-435f-87bf-935a6056e8b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde015c2-b427-435f-87bf-935a6056e8b9-istiod-ca-cert\") pod \"cde015c2-b427-435f-87bf-935a6056e8b9\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " Oct 30 18:41:02.142928 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.142895 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-credential-socket\") pod \"cde015c2-b427-435f-87bf-935a6056e8b9\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " Oct 30 18:41:02.143216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.142927 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-socket\") pod \"cde015c2-b427-435f-87bf-935a6056e8b9\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " Oct 30 18:41:02.143216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.143003 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-envoy\") pod \"0789a37e-7327-47e9-9bc1-247533139ae4\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " Oct 30 18:41:02.143216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.143050 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-data\") pod \"0789a37e-7327-47e9-9bc1-247533139ae4\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " Oct 30 18:41:02.143216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.143087 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0789a37e-7327-47e9-9bc1-247533139ae4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0789a37e-7327-47e9-9bc1-247533139ae4-istio-podinfo\") pod \"0789a37e-7327-47e9-9bc1-247533139ae4\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " Oct 30 18:41:02.143216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.143113 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0789a37e-7327-47e9-9bc1-247533139ae4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0789a37e-7327-47e9-9bc1-247533139ae4-istiod-ca-cert\") pod \"0789a37e-7327-47e9-9bc1-247533139ae4\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " Oct 30 18:41:02.143216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.143158 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-socket\") pod \"0789a37e-7327-47e9-9bc1-247533139ae4\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " Oct 30 18:41:02.143216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.143190 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-kube-api-access-4trtx\" (UniqueName: \"kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-kube-api-access-4trtx\") pod \"0789a37e-7327-47e9-9bc1-247533139ae4\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " Oct 30 18:41:02.143612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.143216 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-istio-token\" (UniqueName: \"kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-istio-token\") pod \"cde015c2-b427-435f-87bf-935a6056e8b9\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " Oct 30 18:41:02.143612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.143263 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-certs\") pod \"0789a37e-7327-47e9-9bc1-247533139ae4\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " Oct 30 18:41:02.143612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.143300 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-credential-socket\") pod \"0789a37e-7327-47e9-9bc1-247533139ae4\" (UID: \"0789a37e-7327-47e9-9bc1-247533139ae4\") " Oct 30 18:41:02.143612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.143368 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-envoy\") pod \"cde015c2-b427-435f-87bf-935a6056e8b9\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " Oct 30 18:41:02.143612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.143428 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-kube-api-access-clbsv\" (UniqueName: \"kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-kube-api-access-clbsv\") pod \"cde015c2-b427-435f-87bf-935a6056e8b9\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " Oct 30 18:41:02.143612 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.143457 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cde015c2-b427-435f-87bf-935a6056e8b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde015c2-b427-435f-87bf-935a6056e8b9-istio-podinfo\") pod \"cde015c2-b427-435f-87bf-935a6056e8b9\" (UID: \"cde015c2-b427-435f-87bf-935a6056e8b9\") " Oct 30 18:41:02.144250 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.144215 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-credential-socket" pod "cde015c2-b427-435f-87bf-935a6056e8b9" (UID: "cde015c2-b427-435f-87bf-935a6056e8b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.144438 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.144416 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-socket" pod "0789a37e-7327-47e9-9bc1-247533139ae4" (UID: "0789a37e-7327-47e9-9bc1-247533139ae4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.144841 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.144815 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0789a37e-7327-47e9-9bc1-247533139ae4-istiod-ca-cert" pod "0789a37e-7327-47e9-9bc1-247533139ae4" (UID: "0789a37e-7327-47e9-9bc1-247533139ae4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:02.145420 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.145156 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-data" pod "0789a37e-7327-47e9-9bc1-247533139ae4" (UID: "0789a37e-7327-47e9-9bc1-247533139ae4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.145700 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde015c2-b427-435f-87bf-935a6056e8b9-istiod-ca-cert" pod "cde015c2-b427-435f-87bf-935a6056e8b9" (UID: "cde015c2-b427-435f-87bf-935a6056e8b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.146327 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cde015c2-b427-435f-87bf-935a6056e8b9-istio-podinfo" pod "cde015c2-b427-435f-87bf-935a6056e8b9" (UID: "cde015c2-b427-435f-87bf-935a6056e8b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.147037 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-data" pod "cde015c2-b427-435f-87bf-935a6056e8b9" (UID: "cde015c2-b427-435f-87bf-935a6056e8b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.147343 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-istio-token" pod "0789a37e-7327-47e9-9bc1-247533139ae4" (UID: "0789a37e-7327-47e9-9bc1-247533139ae4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.147607 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-certs" pod "cde015c2-b427-435f-87bf-935a6056e8b9" (UID: "cde015c2-b427-435f-87bf-935a6056e8b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.147917 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0789a37e-7327-47e9-9bc1-247533139ae4-istio-podinfo" pod "0789a37e-7327-47e9-9bc1-247533139ae4" (UID: "0789a37e-7327-47e9-9bc1-247533139ae4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.148153 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-certs" pod "0789a37e-7327-47e9-9bc1-247533139ae4" (UID: "0789a37e-7327-47e9-9bc1-247533139ae4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.149049 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-envoy" pod "0789a37e-7327-47e9-9bc1-247533139ae4" (UID: "0789a37e-7327-47e9-9bc1-247533139ae4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.150019 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-socket" pod "cde015c2-b427-435f-87bf-935a6056e8b9" (UID: "cde015c2-b427-435f-87bf-935a6056e8b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.150094 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-envoy" pod "cde015c2-b427-435f-87bf-935a6056e8b9" (UID: "cde015c2-b427-435f-87bf-935a6056e8b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.150213 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-istio-token" pod "cde015c2-b427-435f-87bf-935a6056e8b9" (UID: "cde015c2-b427-435f-87bf-935a6056e8b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.150271 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-kube-api-access-clbsv" pod "cde015c2-b427-435f-87bf-935a6056e8b9" (UID: "cde015c2-b427-435f-87bf-935a6056e8b9"). InnerVolumeSpecName "kube-api-access-clbsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.150325 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-kube-api-access-4trtx" pod "0789a37e-7327-47e9-9bc1-247533139ae4" (UID: "0789a37e-7327-47e9-9bc1-247533139ae4"). InnerVolumeSpecName "kube-api-access-4trtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.158034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.151019 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-credential-socket" pod "0789a37e-7327-47e9-9bc1-247533139ae4" (UID: "0789a37e-7327-47e9-9bc1-247533139ae4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.187793 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.187769 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:41:02.244688 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.244601 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-socket\") pod \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " Oct 30 18:41:02.244859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.244702 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-data\") pod \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " Oct 30 18:41:02.244859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.244736 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-envoy\") pod \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " Oct 30 18:41:02.244859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.244761 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istiod-ca-cert\") pod \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " Oct 30 18:41:02.244859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.244806 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-podinfo\") pod \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " Oct 30 18:41:02.245113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.244877 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-kube-api-access-s96xh\" (UniqueName: \"kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-kube-api-access-s96xh\") pod \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " Oct 30 18:41:02.245113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.244901 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-socket" pod "b7981a18-b6ed-477f-abaa-39ef9490ec6f" (UID: "b7981a18-b6ed-477f-abaa-39ef9490ec6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.245113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.244944 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-token\" (UniqueName: \"kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-token\") pod \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " Oct 30 18:41:02.245113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.244987 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-certs\") pod \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " Oct 30 18:41:02.245113 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245041 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-credential-socket\") pod \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\" (UID: \"b7981a18-b6ed-477f-abaa-39ef9490ec6f\") " Oct 30 18:41:02.245359 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245279 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istiod-ca-cert" pod "b7981a18-b6ed-477f-abaa-39ef9490ec6f" (UID: "b7981a18-b6ed-477f-abaa-39ef9490ec6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:02.245600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245576 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.245696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245604 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.245696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245619 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde015c2-b427-435f-87bf-935a6056e8b9-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.245696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245632 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.245696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245646 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.245696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245660 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.245696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245674 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.245696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245687 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245700 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0789a37e-7327-47e9-9bc1-247533139ae4-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245714 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0789a37e-7327-47e9-9bc1-247533139ae4-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245728 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245744 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4trtx\" (UniqueName: \"kubernetes.io/projected/0789a37e-7327-47e9-9bc1-247533139ae4-kube-api-access-4trtx\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245760 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245772 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245784 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245783 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-credential-socket" pod "b7981a18-b6ed-477f-abaa-39ef9490ec6f" (UID: "b7981a18-b6ed-477f-abaa-39ef9490ec6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245797 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0789a37e-7327-47e9-9bc1-247533139ae4-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245810 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245823 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clbsv\" (UniqueName: \"kubernetes.io/projected/cde015c2-b427-435f-87bf-935a6056e8b9-kube-api-access-clbsv\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245837 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde015c2-b427-435f-87bf-935a6056e8b9-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245851 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde015c2-b427-435f-87bf-935a6056e8b9-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245820 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-certs" pod "b7981a18-b6ed-477f-abaa-39ef9490ec6f" (UID: "b7981a18-b6ed-477f-abaa-39ef9490ec6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.246076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.245915 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-data" pod "b7981a18-b6ed-477f-abaa-39ef9490ec6f" (UID: "b7981a18-b6ed-477f-abaa-39ef9490ec6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.247599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.247562 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-token" pod "b7981a18-b6ed-477f-abaa-39ef9490ec6f" (UID: "b7981a18-b6ed-477f-abaa-39ef9490ec6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.247599 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.247591 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-envoy" pod "b7981a18-b6ed-477f-abaa-39ef9490ec6f" (UID: "b7981a18-b6ed-477f-abaa-39ef9490ec6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:02.247793 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.247599 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-podinfo" pod "b7981a18-b6ed-477f-abaa-39ef9490ec6f" (UID: "b7981a18-b6ed-477f-abaa-39ef9490ec6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:02.247873 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.247853 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-kube-api-access-s96xh" pod "b7981a18-b6ed-477f-abaa-39ef9490ec6f" (UID: "b7981a18-b6ed-477f-abaa-39ef9490ec6f"). InnerVolumeSpecName "kube-api-access-s96xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:02.347123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.347088 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.347123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.347118 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.347123 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.347127 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.347364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.347136 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s96xh\" (UniqueName: \"kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-kube-api-access-s96xh\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.347364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.347146 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7981a18-b6ed-477f-abaa-39ef9490ec6f-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.347364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.347156 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.347364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.347164 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7981a18-b6ed-477f-abaa-39ef9490ec6f-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:02.514650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.514575 4883 generic.go:356] "Generic (PLEG): container finished" podID="cde015c2-b427-435f-87bf-935a6056e8b9" containerID="105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2" exitCode=0 Oct 30 18:41:02.514650 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.514619 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" Oct 30 18:41:02.515334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.514655 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" event={"ID":"cde015c2-b427-435f-87bf-935a6056e8b9","Type":"ContainerDied","Data":"105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2"} Oct 30 18:41:02.515334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.514707 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f" event={"ID":"cde015c2-b427-435f-87bf-935a6056e8b9","Type":"ContainerDied","Data":"781495c5b1b865502e072c138569f3deb2a3e89c3fdbd2e879ea6b4402c6d378"} Oct 30 18:41:02.515334 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.514737 4883 scope.go:122] "RemoveContainer" containerID="105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2" Oct 30 18:41:02.520150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.520122 4883 generic.go:356] "Generic (PLEG): container finished" podID="4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7" containerID="be206ca6bb7dfac284e81869b7a9b206816729ae22e4f9a9388e0079a0b8a609" exitCode=0 Oct 30 18:41:02.520257 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.520208 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" event={"ID":"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7","Type":"ContainerDied","Data":"be206ca6bb7dfac284e81869b7a9b206816729ae22e4f9a9388e0079a0b8a609"} Oct 30 18:41:02.521794 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.521769 4883 generic.go:356] "Generic (PLEG): container finished" podID="0789a37e-7327-47e9-9bc1-247533139ae4" containerID="5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46" exitCode=0 Oct 30 18:41:02.521895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.521809 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" Oct 30 18:41:02.521895 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.521847 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" event={"ID":"0789a37e-7327-47e9-9bc1-247533139ae4","Type":"ContainerDied","Data":"5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46"} Oct 30 18:41:02.522041 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.521893 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs" event={"ID":"0789a37e-7327-47e9-9bc1-247533139ae4","Type":"ContainerDied","Data":"d1dacc40e1f2ad9aa8c7c924637fe8133c7c780149b75bf850e734db16886b85"} Oct 30 18:41:02.524144 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.524124 4883 generic.go:356] "Generic (PLEG): container finished" podID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerID="3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7" exitCode=0 Oct 30 18:41:02.524244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.524163 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" Oct 30 18:41:02.524244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.524174 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" event={"ID":"b7981a18-b6ed-477f-abaa-39ef9490ec6f","Type":"ContainerDied","Data":"3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7"} Oct 30 18:41:02.524244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.524208 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc" event={"ID":"b7981a18-b6ed-477f-abaa-39ef9490ec6f","Type":"ContainerDied","Data":"d286d7e0b756259723be26ea69ab7d7a22e0efbecea3f1408c9993d980b2b22f"} Oct 30 18:41:02.534068 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.534050 4883 scope.go:122] "RemoveContainer" containerID="105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2" Oct 30 18:41:02.534497 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:41:02.534475 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2\": container with ID starting with 105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2 not found: ID does not exist" containerID="105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2" Oct 30 18:41:02.534587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.534504 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2"} err="failed to get container status \"105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2\": rpc error: code = NotFound desc = could not find container \"105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2\": container with ID starting with 105f2bb07cc0973ecde0face2ed43d72ed6e124f1127ace8bd78c6a0473ac1b2 not found: ID does not exist" Oct 30 18:41:02.534587 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.534520 4883 scope.go:122] "RemoveContainer" containerID="5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46" Oct 30 18:41:02.560490 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.560471 4883 scope.go:122] "RemoveContainer" containerID="5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46" Oct 30 18:41:02.560815 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:41:02.560796 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46\": container with ID starting with 5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46 not found: ID does not exist" containerID="5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46" Oct 30 18:41:02.560863 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.560823 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46"} err="failed to get container status \"5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46\": rpc error: code = NotFound desc = could not find container \"5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46\": container with ID starting with 5cb8962c8a42ee57693d0c22e962f23aa1b8b7d53cb4e3463911b8e0108e7f46 not found: ID does not exist" Oct 30 18:41:02.560863 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.560842 4883 scope.go:122] "RemoveContainer" containerID="3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7" Oct 30 18:41:02.569720 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.569703 4883 scope.go:122] "RemoveContainer" containerID="e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c" Oct 30 18:41:02.578910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.578891 4883 scope.go:122] "RemoveContainer" containerID="3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7" Oct 30 18:41:02.579238 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:41:02.579213 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7\": container with ID starting with 3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7 not found: ID does not exist" containerID="3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7" Oct 30 18:41:02.579301 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.579247 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7"} err="failed to get container status \"3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7\": rpc error: code = NotFound desc = could not find container \"3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7\": container with ID starting with 3e56d7e936d049b1a4c938a314fa3cee84529815d97329dbf43ca036bf56f1b7 not found: ID does not exist" Oct 30 18:41:02.579301 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.579266 4883 scope.go:122] "RemoveContainer" containerID="e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c" Oct 30 18:41:02.579496 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:41:02.579481 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c\": container with ID starting with e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c not found: ID does not exist" containerID="e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c" Oct 30 18:41:02.579544 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.579500 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c"} err="failed to get container status \"e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c\": rpc error: code = NotFound desc = could not find container \"e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c\": container with ID starting with e829bc8cedacc79d174fe3c1b5dffff56c064dbe9cf8fa9f6bc2e3b887a5053c not found: ID does not exist" Oct 30 18:41:02.587241 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.587211 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f"] Oct 30 18:41:02.600205 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.600184 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-khp9f"] Oct 30 18:41:02.690431 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.690402 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs"] Oct 30 18:41:02.704153 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.704121 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-2t8xs"] Oct 30 18:41:02.761262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.761210 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc"] Oct 30 18:41:02.762124 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:02.762079 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-419d32-openshift-default-6d8ff84c79-4cvwc"] Oct 30 18:41:03.177150 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.177074 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" path="/var/lib/kubelet/pods/0789a37e-7327-47e9-9bc1-247533139ae4/volumes" Oct 30 18:41:03.177579 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.177565 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" path="/var/lib/kubelet/pods/b7981a18-b6ed-477f-abaa-39ef9490ec6f/volumes" Oct 30 18:41:03.178052 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.178037 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" path="/var/lib/kubelet/pods/cde015c2-b427-435f-87bf-935a6056e8b9/volumes" Oct 30 18:41:03.675877 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.675851 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:03.764041 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.764008 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-xks-ca\") pod \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\" (UID: \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\") " Oct 30 18:41:03.764201 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.764054 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-kube-api-access-4klst\" (UniqueName: \"kubernetes.io/projected/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-kube-api-access-4klst\") pod \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\" (UID: \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\") " Oct 30 18:41:03.764201 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.764149 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-openshift-ca\") pod \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\" (UID: \"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7\") " Oct 30 18:41:03.764529 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.764503 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-openshift-ca" pod "4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7" (UID: "4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:03.766191 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.766161 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-xks-ca" pod "4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7" (UID: "4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:03.766283 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.766242 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-kube-api-access-4klst" pod "4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7" (UID: "4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7"). InnerVolumeSpecName "kube-api-access-4klst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:03.865427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.865398 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:03.865427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.865425 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:03.865635 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:03.865435 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4klst\" (UniqueName: \"kubernetes.io/projected/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7-kube-api-access-4klst\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:04.540675 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:04.540645 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" Oct 30 18:41:04.540857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:04.540682 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5" event={"ID":"4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7","Type":"ContainerDied","Data":"5e8f2833cb8cff6f632fad2478c9566933e54ff6ad6aad11ca2bc722206a5bff"} Oct 30 18:41:04.540857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:04.540733 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e8f2833cb8cff6f632fad2478c9566933e54ff6ad6aad11ca2bc722206a5bff" Oct 30 18:41:05.077142 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:41:05.077103 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:41:14.130320 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130284 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm"] Oct 30 18:41:14.130711 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130695 4883 cpu_manager.go:414] "containerMap: removing container" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" Oct 30 18:41:14.130711 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130706 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" Oct 30 18:41:14.130792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130717 4883 cpu_manager.go:414] "containerMap: removing container" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" Oct 30 18:41:14.130792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130723 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" Oct 30 18:41:14.130792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130737 4883 cpu_manager.go:414] "containerMap: removing container" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" Oct 30 18:41:14.130792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130743 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" Oct 30 18:41:14.130792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130757 4883 cpu_manager.go:414] "containerMap: removing container" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" Oct 30 18:41:14.130792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130762 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" Oct 30 18:41:14.130792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130769 4883 cpu_manager.go:414] "containerMap: removing container" podUID="4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7" containerName="revoke-keys" Oct 30 18:41:14.130792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130773 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7" containerName="revoke-keys" Oct 30 18:41:14.131146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130847 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" Oct 30 18:41:14.131146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130856 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="0789a37e-7327-47e9-9bc1-247533139ae4" containerName="istio-proxy" Oct 30 18:41:14.131146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130865 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7981a18-b6ed-477f-abaa-39ef9490ec6f" containerName="istio-proxy" Oct 30 18:41:14.131146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130872 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7" containerName="revoke-keys" Oct 30 18:41:14.131146 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.130882 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde015c2-b427-435f-87bf-935a6056e8b9" containerName="istio-proxy" Oct 30 18:41:14.135343 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.135322 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:14.153284 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.153262 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:41:14.156229 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.156199 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm"] Oct 30 18:41:14.166811 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.166782 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:41:14.167045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.166793 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:41:14.167045 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.166841 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:41:14.170398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.170370 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88939678-hkrfm\" (UID: \"70393036-41e6-4ede-92b9-59eb457bb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:14.170584 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.170442 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d74f2\" (UniqueName: \"kubernetes.io/projected/70393036-41e6-4ede-92b9-59eb457bb4c5-kube-api-access-d74f2\") pod \"maas-api-revoke-sub-88939678-hkrfm\" (UID: \"70393036-41e6-4ede-92b9-59eb457bb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:14.170691 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.170665 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88939678-hkrfm\" (UID: \"70393036-41e6-4ede-92b9-59eb457bb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:14.271854 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.271799 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d74f2\" (UniqueName: \"kubernetes.io/projected/70393036-41e6-4ede-92b9-59eb457bb4c5-kube-api-access-d74f2\") pod \"maas-api-revoke-sub-88939678-hkrfm\" (UID: \"70393036-41e6-4ede-92b9-59eb457bb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:14.272234 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.272031 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88939678-hkrfm\" (UID: \"70393036-41e6-4ede-92b9-59eb457bb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:14.272234 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.272064 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88939678-hkrfm\" (UID: \"70393036-41e6-4ede-92b9-59eb457bb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:14.272753 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.272730 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88939678-hkrfm\" (UID: \"70393036-41e6-4ede-92b9-59eb457bb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:14.274721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.274699 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88939678-hkrfm\" (UID: \"70393036-41e6-4ede-92b9-59eb457bb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:14.297586 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.297552 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d74f2\" (UniqueName: \"kubernetes.io/projected/70393036-41e6-4ede-92b9-59eb457bb4c5-kube-api-access-d74f2\") pod \"maas-api-revoke-sub-88939678-hkrfm\" (UID: \"70393036-41e6-4ede-92b9-59eb457bb4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:14.449553 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.449462 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:14.753216 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:14.753185 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm"] Oct 30 18:41:14.755067 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:41:14.755038 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70393036_41e6_4ede_92b9_59eb457bb4c5.slice/crio-ab008b5671ab3ac79874999b42d1c7f2f1930e8c8bcf915c739f21486c127aef WatchSource:0}: Error finding container ab008b5671ab3ac79874999b42d1c7f2f1930e8c8bcf915c739f21486c127aef: Status 404 returned error can't find the container with id ab008b5671ab3ac79874999b42d1c7f2f1930e8c8bcf915c739f21486c127aef Oct 30 18:41:15.611160 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:15.611131 4883 generic.go:356] "Generic (PLEG): container finished" podID="70393036-41e6-4ede-92b9-59eb457bb4c5" containerID="6a478441ff5e4768ae7b2848e0ba39a63c452c8656b53732aa2cc707b063e312" exitCode=0 Oct 30 18:41:15.611465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:15.611209 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" event={"ID":"70393036-41e6-4ede-92b9-59eb457bb4c5","Type":"ContainerDied","Data":"6a478441ff5e4768ae7b2848e0ba39a63c452c8656b53732aa2cc707b063e312"} Oct 30 18:41:15.611465 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:15.611263 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" event={"ID":"70393036-41e6-4ede-92b9-59eb457bb4c5","Type":"ContainerStarted","Data":"ab008b5671ab3ac79874999b42d1c7f2f1930e8c8bcf915c739f21486c127aef"} Oct 30 18:41:16.758452 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:16.758418 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:16.797345 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:16.797312 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-xks-ca\") pod \"70393036-41e6-4ede-92b9-59eb457bb4c5\" (UID: \"70393036-41e6-4ede-92b9-59eb457bb4c5\") " Oct 30 18:41:16.797531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:16.797372 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-openshift-ca\") pod \"70393036-41e6-4ede-92b9-59eb457bb4c5\" (UID: \"70393036-41e6-4ede-92b9-59eb457bb4c5\") " Oct 30 18:41:16.797531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:16.797415 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70393036-41e6-4ede-92b9-59eb457bb4c5-kube-api-access-d74f2\" (UniqueName: \"kubernetes.io/projected/70393036-41e6-4ede-92b9-59eb457bb4c5-kube-api-access-d74f2\") pod \"70393036-41e6-4ede-92b9-59eb457bb4c5\" (UID: \"70393036-41e6-4ede-92b9-59eb457bb4c5\") " Oct 30 18:41:16.797766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:16.797743 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-openshift-ca" pod "70393036-41e6-4ede-92b9-59eb457bb4c5" (UID: "70393036-41e6-4ede-92b9-59eb457bb4c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:16.799574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:16.799549 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70393036-41e6-4ede-92b9-59eb457bb4c5-kube-api-access-d74f2" pod "70393036-41e6-4ede-92b9-59eb457bb4c5" (UID: "70393036-41e6-4ede-92b9-59eb457bb4c5"). InnerVolumeSpecName "kube-api-access-d74f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:16.799668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:16.799570 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-xks-ca" pod "70393036-41e6-4ede-92b9-59eb457bb4c5" (UID: "70393036-41e6-4ede-92b9-59eb457bb4c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:16.899269 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:16.899180 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:16.899269 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:16.899213 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d74f2\" (UniqueName: \"kubernetes.io/projected/70393036-41e6-4ede-92b9-59eb457bb4c5-kube-api-access-d74f2\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:16.899269 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:16.899224 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70393036-41e6-4ede-92b9-59eb457bb4c5-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:41:17.625700 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:17.625667 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" Oct 30 18:41:17.625923 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:17.625706 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm" event={"ID":"70393036-41e6-4ede-92b9-59eb457bb4c5","Type":"ContainerDied","Data":"ab008b5671ab3ac79874999b42d1c7f2f1930e8c8bcf915c739f21486c127aef"} Oct 30 18:41:17.625923 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:17.625759 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab008b5671ab3ac79874999b42d1c7f2f1930e8c8bcf915c739f21486c127aef" Oct 30 18:41:34.419663 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:34.419627 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5"] Oct 30 18:41:34.432957 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:34.432909 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ac422f-mzdj5"] Oct 30 18:41:35.078256 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:41:35.078220 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:41:35.177026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:41:35.176996 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7" path="/var/lib/kubelet/pods/4eb7a3a9-2369-4dbb-b045-a75ebe87fdb7/volumes" Oct 30 18:42:05.078626 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:42:05.078586 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:42:22.048505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:22.048471 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:42:22.050060 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:22.050025 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:42:22.056939 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:22.056916 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:42:22.058096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:22.058078 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6zpc_d4c590f5-af8b-4994-8b03-7f5cee045767/ovn-acl-logging/0.log" Oct 30 18:42:24.506949 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:24.506900 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6"] Oct 30 18:42:24.507470 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:24.507176 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" podUID="5a60d9c1-369e-4818-8e6b-80e5bd537d36" containerName="istio-proxy" containerID="cri-o://6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d" gracePeriod=30 Oct 30 18:42:25.120859 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.120810 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn"] Oct 30 18:42:25.121371 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.121350 4883 cpu_manager.go:414] "containerMap: removing container" podUID="70393036-41e6-4ede-92b9-59eb457bb4c5" containerName="revoke-keys" Oct 30 18:42:25.121483 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.121372 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70393036-41e6-4ede-92b9-59eb457bb4c5" containerName="revoke-keys" Oct 30 18:42:25.121550 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.121524 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="70393036-41e6-4ede-92b9-59eb457bb4c5" containerName="revoke-keys" Oct 30 18:42:25.124635 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.124612 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:25.140787 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.140764 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:42:25.154285 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.154255 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:42:25.158163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.158129 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn"] Oct 30 18:42:25.174759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.174726 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttr6\" (UniqueName: \"kubernetes.io/projected/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-kube-api-access-kttr6\") pod \"maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn\" (UID: \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:25.174964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.174860 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn\" (UID: \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:25.174964 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.174900 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn\" (UID: \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:25.217041 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.217009 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:42:25.217267 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.217084 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:42:25.276492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.276454 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn\" (UID: \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:25.276492 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.276496 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn\" (UID: \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:25.276746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.276569 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttr6\" (UniqueName: \"kubernetes.io/projected/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-kube-api-access-kttr6\") pod \"maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn\" (UID: \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:25.277233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.277206 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn\" (UID: \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:25.278897 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.278878 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn\" (UID: \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:25.301133 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.301101 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttr6\" (UniqueName: \"kubernetes.io/projected/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-kube-api-access-kttr6\") pod \"maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn\" (UID: \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:25.438792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.438705 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:25.741089 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:25.741065 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn"] Oct 30 18:42:25.743833 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:42:25.743803 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded67f051_fb3e_45b1_a54c_c8e14b0cb320.slice/crio-98bec43a258280ac782edd1bae75be33e3334e0be17f060a9f78e9a9411c741d WatchSource:0}: Error finding container 98bec43a258280ac782edd1bae75be33e3334e0be17f060a9f78e9a9411c741d: Status 404 returned error can't find the container with id 98bec43a258280ac782edd1bae75be33e3334e0be17f060a9f78e9a9411c741d Oct 30 18:42:26.086530 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:26.086494 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" event={"ID":"ed67f051-fb3e-45b1-a54c-c8e14b0cb320","Type":"ContainerStarted","Data":"a08d890e320136b2f8b27edfbcbe10b5856cac1b83c34c245ccca829541fa8cc"} Oct 30 18:42:26.086689 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:26.086567 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" event={"ID":"ed67f051-fb3e-45b1-a54c-c8e14b0cb320","Type":"ContainerStarted","Data":"98bec43a258280ac782edd1bae75be33e3334e0be17f060a9f78e9a9411c741d"} Oct 30 18:42:26.131898 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:26.131754 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" podStartSLOduration=1.131737559 podStartE2EDuration="1.131737559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:26.130123749 +0000 UTC m=+3005.087127585" watchObservedRunningTime="2026-10-30 18:42:26.131737559 +0000 UTC m=+3005.088741394" Oct 30 18:42:27.094514 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:27.094480 4883 generic.go:356] "Generic (PLEG): container finished" podID="ed67f051-fb3e-45b1-a54c-c8e14b0cb320" containerID="a08d890e320136b2f8b27edfbcbe10b5856cac1b83c34c245ccca829541fa8cc" exitCode=0 Oct 30 18:42:27.094914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:27.094560 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" event={"ID":"ed67f051-fb3e-45b1-a54c-c8e14b0cb320","Type":"ContainerDied","Data":"a08d890e320136b2f8b27edfbcbe10b5856cac1b83c34c245ccca829541fa8cc"} Oct 30 18:42:28.286308 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:28.286275 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:28.305307 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:28.305268 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-openshift-ca\") pod \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\" (UID: \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\") " Oct 30 18:42:28.305518 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:28.305406 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-kube-api-access-kttr6\" (UniqueName: \"kubernetes.io/projected/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-kube-api-access-kttr6\") pod \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\" (UID: \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\") " Oct 30 18:42:28.305518 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:28.305471 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-xks-ca\") pod \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\" (UID: \"ed67f051-fb3e-45b1-a54c-c8e14b0cb320\") " Oct 30 18:42:28.305724 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:28.305701 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-openshift-ca" pod "ed67f051-fb3e-45b1-a54c-c8e14b0cb320" (UID: "ed67f051-fb3e-45b1-a54c-c8e14b0cb320"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:28.305839 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:28.305819 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:42:28.308379 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:28.308332 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-kube-api-access-kttr6" pod "ed67f051-fb3e-45b1-a54c-c8e14b0cb320" (UID: "ed67f051-fb3e-45b1-a54c-c8e14b0cb320"). InnerVolumeSpecName "kube-api-access-kttr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:28.308503 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:28.308464 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-xks-ca" pod "ed67f051-fb3e-45b1-a54c-c8e14b0cb320" (UID: "ed67f051-fb3e-45b1-a54c-c8e14b0cb320"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:28.407163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:28.407071 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kttr6\" (UniqueName: \"kubernetes.io/projected/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-kube-api-access-kttr6\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:42:28.407163 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:28.407100 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed67f051-fb3e-45b1-a54c-c8e14b0cb320-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.149466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.149432 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" Oct 30 18:42:29.149466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.149463 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn" event={"ID":"ed67f051-fb3e-45b1-a54c-c8e14b0cb320","Type":"ContainerDied","Data":"98bec43a258280ac782edd1bae75be33e3334e0be17f060a9f78e9a9411c741d"} Oct 30 18:42:29.149781 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.149525 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98bec43a258280ac782edd1bae75be33e3334e0be17f060a9f78e9a9411c741d" Oct 30 18:42:29.778152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.778126 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:42:29.855506 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.855473 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-certs\") pod \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " Oct 30 18:42:29.855718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.855525 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-podinfo\") pod \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " Oct 30 18:42:29.855718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.855588 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istiod-ca-cert\") pod \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " Oct 30 18:42:29.855718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.855622 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-socket\") pod \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " Oct 30 18:42:29.855718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.855651 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-token\" (UniqueName: \"kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-token\") pod \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " Oct 30 18:42:29.855718 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.855697 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-credential-socket\") pod \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " Oct 30 18:42:29.856033 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.855762 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-kube-api-access-f4n8s\" (UniqueName: \"kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-kube-api-access-f4n8s\") pod \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " Oct 30 18:42:29.856033 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.855793 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-data\") pod \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " Oct 30 18:42:29.856033 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.855868 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-envoy\") pod \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\" (UID: \"5a60d9c1-369e-4818-8e6b-80e5bd537d36\") " Oct 30 18:42:29.856296 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.856268 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-certs" pod "5a60d9c1-369e-4818-8e6b-80e5bd537d36" (UID: "5a60d9c1-369e-4818-8e6b-80e5bd537d36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.856521 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.856493 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-credential-socket" pod "5a60d9c1-369e-4818-8e6b-80e5bd537d36" (UID: "5a60d9c1-369e-4818-8e6b-80e5bd537d36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.857185 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.856917 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istiod-ca-cert" pod "5a60d9c1-369e-4818-8e6b-80e5bd537d36" (UID: "5a60d9c1-369e-4818-8e6b-80e5bd537d36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:29.857410 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.857358 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-data" pod "5a60d9c1-369e-4818-8e6b-80e5bd537d36" (UID: "5a60d9c1-369e-4818-8e6b-80e5bd537d36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.858061 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.858039 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-socket" pod "5a60d9c1-369e-4818-8e6b-80e5bd537d36" (UID: "5a60d9c1-369e-4818-8e6b-80e5bd537d36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.858890 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.858858 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-podinfo" pod "5a60d9c1-369e-4818-8e6b-80e5bd537d36" (UID: "5a60d9c1-369e-4818-8e6b-80e5bd537d36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:42:29.860823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.860802 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.860921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.860827 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.860921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.860841 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.860921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.860860 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.860921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.860873 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.860921 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.860888 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.861987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.861732 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-kube-api-access-f4n8s" pod "5a60d9c1-369e-4818-8e6b-80e5bd537d36" (UID: "5a60d9c1-369e-4818-8e6b-80e5bd537d36"). InnerVolumeSpecName "kube-api-access-f4n8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:29.861987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.861745 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-token" pod "5a60d9c1-369e-4818-8e6b-80e5bd537d36" (UID: "5a60d9c1-369e-4818-8e6b-80e5bd537d36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:29.863167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.863141 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-envoy" pod "5a60d9c1-369e-4818-8e6b-80e5bd537d36" (UID: "5a60d9c1-369e-4818-8e6b-80e5bd537d36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:29.962356 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.962320 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.962356 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.962352 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4n8s\" (UniqueName: \"kubernetes.io/projected/5a60d9c1-369e-4818-8e6b-80e5bd537d36-kube-api-access-f4n8s\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:42:29.962356 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:29.962362 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a60d9c1-369e-4818-8e6b-80e5bd537d36-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:42:30.156441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:30.156354 4883 generic.go:356] "Generic (PLEG): container finished" podID="5a60d9c1-369e-4818-8e6b-80e5bd537d36" containerID="6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d" exitCode=0 Oct 30 18:42:30.156441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:30.156415 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" Oct 30 18:42:30.156441 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:30.156424 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" event={"ID":"5a60d9c1-369e-4818-8e6b-80e5bd537d36","Type":"ContainerDied","Data":"6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d"} Oct 30 18:42:30.156675 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:30.156492 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6" event={"ID":"5a60d9c1-369e-4818-8e6b-80e5bd537d36","Type":"ContainerDied","Data":"76c0dcff6a088f26b33fd789b6bf8f68d98a5022c6d00d893a394297642bde43"} Oct 30 18:42:30.156675 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:30.156513 4883 scope.go:122] "RemoveContainer" containerID="6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d" Oct 30 18:42:30.166832 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:30.166808 4883 scope.go:122] "RemoveContainer" containerID="6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d" Oct 30 18:42:30.167181 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:42:30.167161 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d\": container with ID starting with 6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d not found: ID does not exist" containerID="6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d" Oct 30 18:42:30.167268 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:30.167191 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d"} err="failed to get container status \"6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d\": rpc error: code = NotFound desc = could not find container \"6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d\": container with ID starting with 6086d7e2f0d8ffd5ad01d769d14fef9dc71d3022cf87548de9efdca42e5cb94d not found: ID does not exist" Oct 30 18:42:30.213636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:30.213598 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6"] Oct 30 18:42:30.226767 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:30.226725 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ac422f-openshift-default-5cb78544d9-5gcv6"] Oct 30 18:42:31.177318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:31.177285 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a60d9c1-369e-4818-8e6b-80e5bd537d36" path="/var/lib/kubelet/pods/5a60d9c1-369e-4818-8e6b-80e5bd537d36/volumes" Oct 30 18:42:33.493231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:33.493157 4883 scope.go:122] "RemoveContainer" containerID="ee0819f6984ccd1758986bea1617dbe182c547a4918bb45c3aff6ee628a1b6d5" Oct 30 18:42:33.502548 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:33.502529 4883 scope.go:122] "RemoveContainer" containerID="40518a3a1321c0d4d0ce479b1473ec942f6dcc95a4698431b955c20ec0362c65" Oct 30 18:42:33.511376 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:33.511350 4883 scope.go:122] "RemoveContainer" containerID="3b51d51cbf0b98396d4d0fab14434151ab4a56190598c67ec9d9377b1f398a08" Oct 30 18:42:35.081250 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:42:35.081067 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:42:40.528300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:40.528267 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn"] Oct 30 18:42:40.541158 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:40.541120 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-293234-mpjdn"] Oct 30 18:42:41.179966 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:42:41.179914 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed67f051-fb3e-45b1-a54c-c8e14b0cb320" path="/var/lib/kubelet/pods/ed67f051-fb3e-45b1-a54c-c8e14b0cb320/volumes" Oct 30 18:43:03.034524 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:03.034487 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r"] Oct 30 18:43:03.047805 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:03.047768 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1f92be-9mk4r"] Oct 30 18:43:03.049534 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:03.049521 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj"] Oct 30 18:43:03.051175 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:03.051155 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-044a8b97-cjpfj"] Oct 30 18:43:03.180552 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:03.180519 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f9a074-8927-4f97-9287-6105c91e6bfb" path="/var/lib/kubelet/pods/64f9a074-8927-4f97-9287-6105c91e6bfb/volumes" Oct 30 18:43:03.181108 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:03.181083 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a3efb9-7e77-4ed8-a69b-25f7d1446b19" path="/var/lib/kubelet/pods/70a3efb9-7e77-4ed8-a69b-25f7d1446b19/volumes" Oct 30 18:43:05.078349 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:43:05.078318 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:43:16.032385 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:16.032341 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd"] Oct 30 18:43:16.045790 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:16.045757 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx"] Oct 30 18:43:16.047679 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:16.047658 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c3ebfb-tt6wd"] Oct 30 18:43:16.049348 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:16.049321 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744af70c-2xgtx"] Oct 30 18:43:17.177861 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:17.177827 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16bf630-ad79-492f-ad96-cf2cdfad0bb3" path="/var/lib/kubelet/pods/b16bf630-ad79-492f-ad96-cf2cdfad0bb3/volumes" Oct 30 18:43:17.178303 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:17.178288 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c263b264-7bcd-4bcb-a8b1-f81d46374867" path="/var/lib/kubelet/pods/c263b264-7bcd-4bcb-a8b1-f81d46374867/volumes" Oct 30 18:43:25.803094 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:25.803056 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr"] Oct 30 18:43:25.805575 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:25.803266 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" podUID="a49a4773-1070-4484-9894-32d306f96e59" containerName="discovery" containerID="cri-o://4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f" gracePeriod=30 Oct 30 18:43:26.062092 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.062026 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:43:26.156026 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.155992 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49a4773-1070-4484-9894-32d306f96e59-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49a4773-1070-4484-9894-32d306f96e59-local-certs\") pod \"a49a4773-1070-4484-9894-32d306f96e59\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " Oct 30 18:43:26.156243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.156070 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-istio-token\" (UniqueName: \"kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-istio-token\") pod \"a49a4773-1070-4484-9894-32d306f96e59\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " Oct 30 18:43:26.156243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.156110 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-kube-api-access-tb622\" (UniqueName: \"kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-kube-api-access-tb622\") pod \"a49a4773-1070-4484-9894-32d306f96e59\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " Oct 30 18:43:26.156243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.156178 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-dns-cert\") pod \"a49a4773-1070-4484-9894-32d306f96e59\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " Oct 30 18:43:26.156243 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.156231 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-ca-configmap\") pod \"a49a4773-1070-4484-9894-32d306f96e59\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " Oct 30 18:43:26.156487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.156276 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-cacerts\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-cacerts\") pod \"a49a4773-1070-4484-9894-32d306f96e59\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " Oct 30 18:43:26.156487 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.156305 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-kubeconfig\") pod \"a49a4773-1070-4484-9894-32d306f96e59\" (UID: \"a49a4773-1070-4484-9894-32d306f96e59\") " Oct 30 18:43:26.156815 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.156765 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-ca-configmap" pod "a49a4773-1070-4484-9894-32d306f96e59" (UID: "a49a4773-1070-4484-9894-32d306f96e59"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:26.158439 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.158410 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49a4773-1070-4484-9894-32d306f96e59-local-certs" pod "a49a4773-1070-4484-9894-32d306f96e59" (UID: "a49a4773-1070-4484-9894-32d306f96e59"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:26.158549 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.158446 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-dns-cert" pod "a49a4773-1070-4484-9894-32d306f96e59" (UID: "a49a4773-1070-4484-9894-32d306f96e59"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:26.158717 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.158699 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-istio-token" pod "a49a4773-1070-4484-9894-32d306f96e59" (UID: "a49a4773-1070-4484-9894-32d306f96e59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:26.158783 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.158737 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-kube-api-access-tb622" pod "a49a4773-1070-4484-9894-32d306f96e59" (UID: "a49a4773-1070-4484-9894-32d306f96e59"). InnerVolumeSpecName "kube-api-access-tb622". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:26.158914 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.158893 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-kubeconfig" pod "a49a4773-1070-4484-9894-32d306f96e59" (UID: "a49a4773-1070-4484-9894-32d306f96e59"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:26.159058 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.159030 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-cacerts" pod "a49a4773-1070-4484-9894-32d306f96e59" (UID: "a49a4773-1070-4484-9894-32d306f96e59"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:26.257919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.257882 4883 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49a4773-1070-4484-9894-32d306f96e59-local-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:43:26.257919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.257911 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:43:26.257919 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.257924 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb622\" (UniqueName: \"kubernetes.io/projected/a49a4773-1070-4484-9894-32d306f96e59-kube-api-access-tb622\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:43:26.258306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.257964 4883 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-dns-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:43:26.258306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.257978 4883 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a49a4773-1070-4484-9894-32d306f96e59-istio-csr-ca-configmap\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:43:26.258306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.257989 4883 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-cacerts\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:43:26.258306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.258001 4883 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a49a4773-1070-4484-9894-32d306f96e59-istio-kubeconfig\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:43:26.589349 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.589309 4883 generic.go:356] "Generic (PLEG): container finished" podID="a49a4773-1070-4484-9894-32d306f96e59" containerID="4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f" exitCode=0 Oct 30 18:43:26.589537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.589362 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" Oct 30 18:43:26.589537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.589390 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" event={"ID":"a49a4773-1070-4484-9894-32d306f96e59","Type":"ContainerDied","Data":"4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f"} Oct 30 18:43:26.589537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.589453 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr" event={"ID":"a49a4773-1070-4484-9894-32d306f96e59","Type":"ContainerDied","Data":"4f82a49ab009f8bdea05f2d219d967720effb4790183e65cb3c2a81255fe5133"} Oct 30 18:43:26.589537 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.589478 4883 scope.go:122] "RemoveContainer" containerID="4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f" Oct 30 18:43:26.601025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.601003 4883 scope.go:122] "RemoveContainer" containerID="4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f" Oct 30 18:43:26.601299 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:43:26.601277 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f\": container with ID starting with 4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f not found: ID does not exist" containerID="4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f" Oct 30 18:43:26.601357 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.601311 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f"} err="failed to get container status \"4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f\": rpc error: code = NotFound desc = could not find container \"4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f\": container with ID starting with 4c8dd0a9a4ba41a68a2f2b96f4f2b98def8082accb62e4af8b2b91f36abdf13f not found: ID does not exist" Oct 30 18:43:26.645671 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.645640 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr"] Oct 30 18:43:26.658683 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:26.658650 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47vdr"] Oct 30 18:43:27.177790 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:27.177754 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49a4773-1070-4484-9894-32d306f96e59" path="/var/lib/kubelet/pods/a49a4773-1070-4484-9894-32d306f96e59/volumes" Oct 30 18:43:33.547287 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:33.547253 4883 scope.go:122] "RemoveContainer" containerID="ec7f2709047e35cfc7e5893d3b1d25ce4c1ad3bd6648299cacebb0d82ca39fa5" Oct 30 18:43:33.556629 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:43:33.556604 4883 scope.go:122] "RemoveContainer" containerID="7097c5c4a35fe1ce266f58bdb52bc29d9345fc5e70586527192b777a9d8cbfdc" Oct 30 18:43:35.077841 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:43:35.077803 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:44:00.533494 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:00.533452 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:44:00.534252 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:00.534220 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:44:01.964427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:01.964323 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8"] Oct 30 18:44:01.964980 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:01.964962 4883 cpu_manager.go:414] "containerMap: removing container" podUID="a49a4773-1070-4484-9894-32d306f96e59" containerName="discovery" Oct 30 18:44:01.965076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:01.964982 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49a4773-1070-4484-9894-32d306f96e59" containerName="discovery" Oct 30 18:44:01.965076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:01.965004 4883 cpu_manager.go:414] "containerMap: removing container" podUID="ed67f051-fb3e-45b1-a54c-c8e14b0cb320" containerName="revoke-keys" Oct 30 18:44:01.965076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:01.965012 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed67f051-fb3e-45b1-a54c-c8e14b0cb320" containerName="revoke-keys" Oct 30 18:44:01.965076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:01.965047 4883 cpu_manager.go:414] "containerMap: removing container" podUID="5a60d9c1-369e-4818-8e6b-80e5bd537d36" containerName="istio-proxy" Oct 30 18:44:01.965076 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:01.965055 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a60d9c1-369e-4818-8e6b-80e5bd537d36" containerName="istio-proxy" Oct 30 18:44:01.965341 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:01.965172 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a60d9c1-369e-4818-8e6b-80e5bd537d36" containerName="istio-proxy" Oct 30 18:44:01.965341 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:01.965190 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49a4773-1070-4484-9894-32d306f96e59" containerName="discovery" Oct 30 18:44:01.965341 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:01.965203 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed67f051-fb3e-45b1-a54c-c8e14b0cb320" containerName="revoke-keys" Oct 30 18:44:01.968645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:01.968624 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:02.001232 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.001191 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:44:02.001377 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.001246 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:44:02.001377 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.001262 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:44:02.001377 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.001266 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:44:02.004403 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.004372 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8"] Oct 30 18:44:02.107779 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.107733 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8\" (UID: \"759f35d1-e20b-4795-9c0a-773a4241a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:02.107986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.107847 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gbwz\" (UniqueName: \"kubernetes.io/projected/759f35d1-e20b-4795-9c0a-773a4241a8cd-kube-api-access-6gbwz\") pod \"maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8\" (UID: \"759f35d1-e20b-4795-9c0a-773a4241a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:02.107986 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.107966 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8\" (UID: \"759f35d1-e20b-4795-9c0a-773a4241a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:02.208815 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.208772 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8\" (UID: \"759f35d1-e20b-4795-9c0a-773a4241a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:02.209043 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.208967 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gbwz\" (UniqueName: \"kubernetes.io/projected/759f35d1-e20b-4795-9c0a-773a4241a8cd-kube-api-access-6gbwz\") pod \"maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8\" (UID: \"759f35d1-e20b-4795-9c0a-773a4241a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:02.209128 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.209086 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8\" (UID: \"759f35d1-e20b-4795-9c0a-773a4241a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:02.209545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.209520 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8\" (UID: \"759f35d1-e20b-4795-9c0a-773a4241a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:02.211404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.211383 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8\" (UID: \"759f35d1-e20b-4795-9c0a-773a4241a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:02.233301 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.233248 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gbwz\" (UniqueName: \"kubernetes.io/projected/759f35d1-e20b-4795-9c0a-773a4241a8cd-kube-api-access-6gbwz\") pod \"maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8\" (UID: \"759f35d1-e20b-4795-9c0a-773a4241a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:02.282206 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.282178 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:02.583629 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.583602 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8"] Oct 30 18:44:02.585677 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:44:02.585649 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759f35d1_e20b_4795_9c0a_773a4241a8cd.slice/crio-05d295caed3a2a3e93a73bd18aee8530b0939d70b5f02c6d2615487f2a2f0ec9 WatchSource:0}: Error finding container 05d295caed3a2a3e93a73bd18aee8530b0939d70b5f02c6d2615487f2a2f0ec9: Status 404 returned error can't find the container with id 05d295caed3a2a3e93a73bd18aee8530b0939d70b5f02c6d2615487f2a2f0ec9 Oct 30 18:44:02.853676 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:02.853640 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" event={"ID":"759f35d1-e20b-4795-9c0a-773a4241a8cd","Type":"ContainerStarted","Data":"05d295caed3a2a3e93a73bd18aee8530b0939d70b5f02c6d2615487f2a2f0ec9"} Oct 30 18:44:03.860395 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:03.860364 4883 generic.go:356] "Generic (PLEG): container finished" podID="759f35d1-e20b-4795-9c0a-773a4241a8cd" containerID="e4d8afc492c7eec8bdebd252a401938be5ae40fcba29b2985937082b41dd861c" exitCode=0 Oct 30 18:44:03.860826 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:03.860446 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" event={"ID":"759f35d1-e20b-4795-9c0a-773a4241a8cd","Type":"ContainerDied","Data":"e4d8afc492c7eec8bdebd252a401938be5ae40fcba29b2985937082b41dd861c"} Oct 30 18:44:05.010926 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.010895 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:05.086349 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:44:05.086311 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:44:05.136181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.136092 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-openshift-ca\") pod \"759f35d1-e20b-4795-9c0a-773a4241a8cd\" (UID: \"759f35d1-e20b-4795-9c0a-773a4241a8cd\") " Oct 30 18:44:05.136181 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.136157 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-xks-ca\") pod \"759f35d1-e20b-4795-9c0a-773a4241a8cd\" (UID: \"759f35d1-e20b-4795-9c0a-773a4241a8cd\") " Oct 30 18:44:05.136450 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.136248 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759f35d1-e20b-4795-9c0a-773a4241a8cd-kube-api-access-6gbwz\" (UniqueName: \"kubernetes.io/projected/759f35d1-e20b-4795-9c0a-773a4241a8cd-kube-api-access-6gbwz\") pod \"759f35d1-e20b-4795-9c0a-773a4241a8cd\" (UID: \"759f35d1-e20b-4795-9c0a-773a4241a8cd\") " Oct 30 18:44:05.136555 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.136518 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-openshift-ca" pod "759f35d1-e20b-4795-9c0a-773a4241a8cd" (UID: "759f35d1-e20b-4795-9c0a-773a4241a8cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:05.138288 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.138263 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-xks-ca" pod "759f35d1-e20b-4795-9c0a-773a4241a8cd" (UID: "759f35d1-e20b-4795-9c0a-773a4241a8cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:05.138365 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.138303 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759f35d1-e20b-4795-9c0a-773a4241a8cd-kube-api-access-6gbwz" pod "759f35d1-e20b-4795-9c0a-773a4241a8cd" (UID: "759f35d1-e20b-4795-9c0a-773a4241a8cd"). InnerVolumeSpecName "kube-api-access-6gbwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:05.237228 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.237190 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gbwz\" (UniqueName: \"kubernetes.io/projected/759f35d1-e20b-4795-9c0a-773a4241a8cd-kube-api-access-6gbwz\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:05.237228 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.237221 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:05.237228 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.237232 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/759f35d1-e20b-4795-9c0a-773a4241a8cd-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:05.874885 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.874856 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" Oct 30 18:44:05.875290 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.874908 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8" event={"ID":"759f35d1-e20b-4795-9c0a-773a4241a8cd","Type":"ContainerDied","Data":"05d295caed3a2a3e93a73bd18aee8530b0939d70b5f02c6d2615487f2a2f0ec9"} Oct 30 18:44:05.875290 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:05.874989 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05d295caed3a2a3e93a73bd18aee8530b0939d70b5f02c6d2615487f2a2f0ec9" Oct 30 18:44:06.651137 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:06.651102 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv"] Oct 30 18:44:15.230768 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.230729 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5"] Oct 30 18:44:15.231182 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.231169 4883 cpu_manager.go:414] "containerMap: removing container" podUID="759f35d1-e20b-4795-9c0a-773a4241a8cd" containerName="revoke-keys" Oct 30 18:44:15.231226 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.231181 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759f35d1-e20b-4795-9c0a-773a4241a8cd" containerName="revoke-keys" Oct 30 18:44:15.231262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.231247 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="759f35d1-e20b-4795-9c0a-773a4241a8cd" containerName="revoke-keys" Oct 30 18:44:15.234417 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.234397 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:15.254870 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.254840 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:44:15.255036 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.254925 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:44:15.268679 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.268632 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:44:15.268857 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.268801 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jtbsh\"" Oct 30 18:44:15.271770 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.271743 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5"] Oct 30 18:44:15.365381 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.365347 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4lm\" (UniqueName: \"kubernetes.io/projected/f3342acd-bd0f-4e2e-b346-130a82fc2589-kube-api-access-mp4lm\") pod \"maas-api-revoke-sub-6fd00bab-t6kl5\" (UID: \"f3342acd-bd0f-4e2e-b346-130a82fc2589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:15.365567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.365495 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fd00bab-t6kl5\" (UID: \"f3342acd-bd0f-4e2e-b346-130a82fc2589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:15.365567 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.365523 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fd00bab-t6kl5\" (UID: \"f3342acd-bd0f-4e2e-b346-130a82fc2589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:15.466795 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.466750 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fd00bab-t6kl5\" (UID: \"f3342acd-bd0f-4e2e-b346-130a82fc2589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:15.466795 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.466797 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fd00bab-t6kl5\" (UID: \"f3342acd-bd0f-4e2e-b346-130a82fc2589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:15.467093 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.466850 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4lm\" (UniqueName: \"kubernetes.io/projected/f3342acd-bd0f-4e2e-b346-130a82fc2589-kube-api-access-mp4lm\") pod \"maas-api-revoke-sub-6fd00bab-t6kl5\" (UID: \"f3342acd-bd0f-4e2e-b346-130a82fc2589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:15.467518 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.467490 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fd00bab-t6kl5\" (UID: \"f3342acd-bd0f-4e2e-b346-130a82fc2589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:15.469247 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.469223 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fd00bab-t6kl5\" (UID: \"f3342acd-bd0f-4e2e-b346-130a82fc2589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:15.492034 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.491969 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4lm\" (UniqueName: \"kubernetes.io/projected/f3342acd-bd0f-4e2e-b346-130a82fc2589-kube-api-access-mp4lm\") pod \"maas-api-revoke-sub-6fd00bab-t6kl5\" (UID: \"f3342acd-bd0f-4e2e-b346-130a82fc2589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:15.548025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.547992 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:15.848970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.848922 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5"] Oct 30 18:44:15.851763 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:44:15.851735 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3342acd_bd0f_4e2e_b346_130a82fc2589.slice/crio-ec98e9bf018fe611b576e7fc60585e7bd436ae9256f940b893a63abc74add392 WatchSource:0}: Error finding container ec98e9bf018fe611b576e7fc60585e7bd436ae9256f940b893a63abc74add392: Status 404 returned error can't find the container with id ec98e9bf018fe611b576e7fc60585e7bd436ae9256f940b893a63abc74add392 Oct 30 18:44:15.963198 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:15.963161 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" event={"ID":"f3342acd-bd0f-4e2e-b346-130a82fc2589","Type":"ContainerStarted","Data":"ec98e9bf018fe611b576e7fc60585e7bd436ae9256f940b893a63abc74add392"} Oct 30 18:44:16.971179 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:16.971137 4883 generic.go:356] "Generic (PLEG): container finished" podID="f3342acd-bd0f-4e2e-b346-130a82fc2589" containerID="2d73407712b899a24608819efbc4e235f1e627be250b0cd2d7cba9ce318eb871" exitCode=0 Oct 30 18:44:16.971675 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:16.971220 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" event={"ID":"f3342acd-bd0f-4e2e-b346-130a82fc2589","Type":"ContainerDied","Data":"2d73407712b899a24608819efbc4e235f1e627be250b0cd2d7cba9ce318eb871"} Oct 30 18:44:18.362700 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:18.362674 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:18.530982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:18.530322 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-openshift-ca\") pod \"f3342acd-bd0f-4e2e-b346-130a82fc2589\" (UID: \"f3342acd-bd0f-4e2e-b346-130a82fc2589\") " Oct 30 18:44:18.530982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:18.530426 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3342acd-bd0f-4e2e-b346-130a82fc2589-kube-api-access-mp4lm\" (UniqueName: \"kubernetes.io/projected/f3342acd-bd0f-4e2e-b346-130a82fc2589-kube-api-access-mp4lm\") pod \"f3342acd-bd0f-4e2e-b346-130a82fc2589\" (UID: \"f3342acd-bd0f-4e2e-b346-130a82fc2589\") " Oct 30 18:44:18.530982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:18.530483 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-xks-ca\") pod \"f3342acd-bd0f-4e2e-b346-130a82fc2589\" (UID: \"f3342acd-bd0f-4e2e-b346-130a82fc2589\") " Oct 30 18:44:18.531590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:18.531551 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-openshift-ca" pod "f3342acd-bd0f-4e2e-b346-130a82fc2589" (UID: "f3342acd-bd0f-4e2e-b346-130a82fc2589"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:18.537398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:18.537369 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-xks-ca" pod "f3342acd-bd0f-4e2e-b346-130a82fc2589" (UID: "f3342acd-bd0f-4e2e-b346-130a82fc2589"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:18.537710 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:18.537674 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3342acd-bd0f-4e2e-b346-130a82fc2589-kube-api-access-mp4lm" pod "f3342acd-bd0f-4e2e-b346-130a82fc2589" (UID: "f3342acd-bd0f-4e2e-b346-130a82fc2589"). InnerVolumeSpecName "kube-api-access-mp4lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:18.633710 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:18.633669 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-openshift-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:18.634025 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:18.634004 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4lm\" (UniqueName: \"kubernetes.io/projected/f3342acd-bd0f-4e2e-b346-130a82fc2589-kube-api-access-mp4lm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:18.634170 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:18.634141 4883 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3342acd-bd0f-4e2e-b346-130a82fc2589-maas-api-xks-ca\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:19.000608 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:19.000539 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" Oct 30 18:44:19.000758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:19.000540 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd00bab-t6kl5" event={"ID":"f3342acd-bd0f-4e2e-b346-130a82fc2589","Type":"ContainerDied","Data":"ec98e9bf018fe611b576e7fc60585e7bd436ae9256f940b893a63abc74add392"} Oct 30 18:44:19.000758 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:19.000667 4883 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec98e9bf018fe611b576e7fc60585e7bd436ae9256f940b893a63abc74add392" Oct 30 18:44:21.743592 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:21.743554 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" containerID="cri-o://9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc" gracePeriod=45 Oct 30 18:44:22.004242 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.004176 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:44:22.018929 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.018895 4883 generic.go:356] "Generic (PLEG): container finished" podID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerID="9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc" exitCode=0 Oct 30 18:44:22.018929 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.018951 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" Oct 30 18:44:22.019192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.018982 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" event={"ID":"1ff654c6-de28-43a9-a216-6dc69a9a6752","Type":"ContainerDied","Data":"9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc"} Oct 30 18:44:22.019192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.019031 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv" event={"ID":"1ff654c6-de28-43a9-a216-6dc69a9a6752","Type":"ContainerDied","Data":"f6f07cfa27e44a53b83b04bc53debe52da6ab39b39108af1e50eefe181e2a5e4"} Oct 30 18:44:22.019192 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.019052 4883 scope.go:122] "RemoveContainer" containerID="9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc" Oct 30 18:44:22.028393 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.028373 4883 scope.go:122] "RemoveContainer" containerID="9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc" Oct 30 18:44:22.028701 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:44:22.028681 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc\": container with ID starting with 9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc not found: ID does not exist" containerID="9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc" Oct 30 18:44:22.028751 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.028712 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc"} err="failed to get container status \"9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc\": rpc error: code = NotFound desc = could not find container \"9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc\": container with ID starting with 9d2ae7148d8b51537b346125c515c610b6590b970e708523af67aad679a8fefc not found: ID does not exist" Oct 30 18:44:22.068981 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.068914 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ff654c6-de28-43a9-a216-6dc69a9a6752-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff654c6-de28-43a9-a216-6dc69a9a6752-tls-certs\") pod \"1ff654c6-de28-43a9-a216-6dc69a9a6752\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " Oct 30 18:44:22.069152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.069019 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-home\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-home\") pod \"1ff654c6-de28-43a9-a216-6dc69a9a6752\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " Oct 30 18:44:22.069152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.069081 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-dshm\") pod \"1ff654c6-de28-43a9-a216-6dc69a9a6752\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " Oct 30 18:44:22.069294 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.069148 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-model-cache\") pod \"1ff654c6-de28-43a9-a216-6dc69a9a6752\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " Oct 30 18:44:22.069294 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.069182 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff654c6-de28-43a9-a216-6dc69a9a6752-kube-api-access-7lfxp\" (UniqueName: \"kubernetes.io/projected/1ff654c6-de28-43a9-a216-6dc69a9a6752-kube-api-access-7lfxp\") pod \"1ff654c6-de28-43a9-a216-6dc69a9a6752\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " Oct 30 18:44:22.069294 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.069223 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-tmp-dir\") pod \"1ff654c6-de28-43a9-a216-6dc69a9a6752\" (UID: \"1ff654c6-de28-43a9-a216-6dc69a9a6752\") " Oct 30 18:44:22.069446 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.069295 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-home" pod "1ff654c6-de28-43a9-a216-6dc69a9a6752" (UID: "1ff654c6-de28-43a9-a216-6dc69a9a6752"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.069505 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.069469 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-model-cache" pod "1ff654c6-de28-43a9-a216-6dc69a9a6752" (UID: "1ff654c6-de28-43a9-a216-6dc69a9a6752"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.069591 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.069564 4883 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-home\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.069591 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.069591 4883 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-model-cache\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.069808 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.069694 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-tmp-dir" pod "1ff654c6-de28-43a9-a216-6dc69a9a6752" (UID: "1ff654c6-de28-43a9-a216-6dc69a9a6752"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.071848 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.071821 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-dshm" pod "1ff654c6-de28-43a9-a216-6dc69a9a6752" (UID: "1ff654c6-de28-43a9-a216-6dc69a9a6752"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.072005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.071856 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff654c6-de28-43a9-a216-6dc69a9a6752-tls-certs" pod "1ff654c6-de28-43a9-a216-6dc69a9a6752" (UID: "1ff654c6-de28-43a9-a216-6dc69a9a6752"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:22.072005 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.071884 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff654c6-de28-43a9-a216-6dc69a9a6752-kube-api-access-7lfxp" pod "1ff654c6-de28-43a9-a216-6dc69a9a6752" (UID: "1ff654c6-de28-43a9-a216-6dc69a9a6752"). InnerVolumeSpecName "kube-api-access-7lfxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:22.171140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.171107 4883 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-dshm\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.171140 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.171138 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lfxp\" (UniqueName: \"kubernetes.io/projected/1ff654c6-de28-43a9-a216-6dc69a9a6752-kube-api-access-7lfxp\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.171367 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.171169 4883 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff654c6-de28-43a9-a216-6dc69a9a6752-tmp-dir\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.171367 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.171182 4883 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff654c6-de28-43a9-a216-6dc69a9a6752-tls-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.368732 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.368693 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv"] Oct 30 18:44:22.381911 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:22.381876 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b424a5/premium-w1-b424a5-kserve-68c9d8696c-nlbqv"] Oct 30 18:44:23.178027 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:23.177993 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" path="/var/lib/kubelet/pods/1ff654c6-de28-43a9-a216-6dc69a9a6752/volumes" Oct 30 18:44:31.078997 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:31.078960 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8"] Oct 30 18:44:31.091739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:31.091706 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b424a5-vcqf8"] Oct 30 18:44:31.177042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:31.177007 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="759f35d1-e20b-4795-9c0a-773a4241a8cd" path="/var/lib/kubelet/pods/759f35d1-e20b-4795-9c0a-773a4241a8cd/volumes" Oct 30 18:44:33.591149 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:33.591116 4883 scope.go:122] "RemoveContainer" containerID="3dbf576228dee177478beb748ef557e5f0b9a0a79db8365efdc4b24ecd47ea1d" Oct 30 18:44:33.599844 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:33.599819 4883 scope.go:122] "RemoveContainer" containerID="70d823ba563dec50eb424fe37bdd279ee893bd9f2e6a7c504979b272a46c9555" Oct 30 18:44:33.607982 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:33.607958 4883 scope.go:122] "RemoveContainer" containerID="93203434a0efffc2813b9c124046cbf574a40500ae6df48bc1bf98329c9143e9" Oct 30 18:44:33.615956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:33.615918 4883 scope.go:122] "RemoveContainer" containerID="20a2f17b46487cc0b08c1199b789b708a6f3cb4e6e40323494f9f6e145984fd7" Oct 30 18:44:33.624682 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:33.624658 4883 scope.go:122] "RemoveContainer" containerID="7d33abb86324ba0aaa007fe1c2a012bb7f6de2ac01acbb9db85eb5d9ec266880" Oct 30 18:44:33.632978 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:33.632959 4883 scope.go:122] "RemoveContainer" containerID="f2fda8eb22cb8125cf285777fa2b36b2edee1a627491b39f1f6376c4d2814852" Oct 30 18:44:33.640879 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:33.640853 4883 scope.go:122] "RemoveContainer" containerID="28c134b0304a6694047d20226b990b41ea5ee771602d11d17181d705a2856e4d" Oct 30 18:44:35.077258 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:44:35.077226 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:44:37.858958 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:37.858908 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5"] Oct 30 18:44:37.860987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:37.859201 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" containerID="cri-o://74f73e3788afb15f9fb12fb3b198e70a6a8b6d103eaf4a2637efe694decbd051" gracePeriod=30 Oct 30 18:44:37.888915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:37.888884 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk"] Oct 30 18:44:37.889167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:37.889123 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" containerID="cri-o://45ed41507d39622999d6b7ddbea3fff5d193bf4bdebbf183916bfa2d73f04e8a" gracePeriod=30 Oct 30 18:44:37.892413 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:37.892385 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r"] Oct 30 18:44:37.892620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:37.892595 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" containerID="cri-o://051b3f39ef48e1356b850dc0c578de9b542c6d9c588ae6556a252e8add3edfc3" gracePeriod=30 Oct 30 18:44:39.579548 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:39.579516 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_developer-portal-controller-66d8dbb8c8-d7cds_282b0162-4f29-4198-a4dc-50b21b6266b6/manager/0.log" Oct 30 18:44:42.379877 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:42.379835 4883 patch_prober.go:29] interesting pod/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:44:42.380282 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:42.379927 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:44:43.146776 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.146712 4883 generic.go:356] "Generic (PLEG): container finished" podID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerID="45ed41507d39622999d6b7ddbea3fff5d193bf4bdebbf183916bfa2d73f04e8a" exitCode=0 Oct 30 18:44:43.146971 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.146790 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" event={"ID":"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea","Type":"ContainerDied","Data":"45ed41507d39622999d6b7ddbea3fff5d193bf4bdebbf183916bfa2d73f04e8a"} Oct 30 18:44:43.148639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.148292 4883 generic.go:356] "Generic (PLEG): container finished" podID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerID="74f73e3788afb15f9fb12fb3b198e70a6a8b6d103eaf4a2637efe694decbd051" exitCode=0 Oct 30 18:44:43.148639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.148386 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" event={"ID":"d59db4be-b972-48bc-b2dc-298d5da131dc","Type":"ContainerDied","Data":"74f73e3788afb15f9fb12fb3b198e70a6a8b6d103eaf4a2637efe694decbd051"} Oct 30 18:44:43.150668 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.150651 4883 generic.go:356] "Generic (PLEG): container finished" podID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerID="051b3f39ef48e1356b850dc0c578de9b542c6d9c588ae6556a252e8add3edfc3" exitCode=0 Oct 30 18:44:43.150766 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.150725 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" event={"ID":"357860ec-9d4e-4d5e-be19-eec58835b3dc","Type":"ContainerDied","Data":"051b3f39ef48e1356b850dc0c578de9b542c6d9c588ae6556a252e8add3edfc3"} Oct 30 18:44:43.179600 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.179568 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:44:43.253555 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.253522 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:44:43.257328 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.257302 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:44:43.302452 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.302413 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-credential-socket\") pod \"d59db4be-b972-48bc-b2dc-298d5da131dc\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " Oct 30 18:44:43.302636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.302469 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-certs\") pod \"d59db4be-b972-48bc-b2dc-298d5da131dc\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " Oct 30 18:44:43.302636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.302521 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-podinfo\") pod \"d59db4be-b972-48bc-b2dc-298d5da131dc\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " Oct 30 18:44:43.302636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.302567 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-socket\") pod \"d59db4be-b972-48bc-b2dc-298d5da131dc\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " Oct 30 18:44:43.302636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.302603 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-envoy\") pod \"d59db4be-b972-48bc-b2dc-298d5da131dc\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " Oct 30 18:44:43.302636 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.302619 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-data\") pod \"d59db4be-b972-48bc-b2dc-298d5da131dc\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " Oct 30 18:44:43.302899 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.302639 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-token\" (UniqueName: \"kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-token\") pod \"d59db4be-b972-48bc-b2dc-298d5da131dc\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " Oct 30 18:44:43.302899 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.302673 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-kube-api-access-nbqdp\" (UniqueName: \"kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-kube-api-access-nbqdp\") pod \"d59db4be-b972-48bc-b2dc-298d5da131dc\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " Oct 30 18:44:43.302899 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.302734 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59db4be-b972-48bc-b2dc-298d5da131dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59db4be-b972-48bc-b2dc-298d5da131dc-istiod-ca-cert\") pod \"d59db4be-b972-48bc-b2dc-298d5da131dc\" (UID: \"d59db4be-b972-48bc-b2dc-298d5da131dc\") " Oct 30 18:44:43.303101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.303020 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-socket" pod "d59db4be-b972-48bc-b2dc-298d5da131dc" (UID: "d59db4be-b972-48bc-b2dc-298d5da131dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.303101 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.303036 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-data" pod "d59db4be-b972-48bc-b2dc-298d5da131dc" (UID: "d59db4be-b972-48bc-b2dc-298d5da131dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.303299 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.303277 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-credential-socket" pod "d59db4be-b972-48bc-b2dc-298d5da131dc" (UID: "d59db4be-b972-48bc-b2dc-298d5da131dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.303463 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.303287 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59db4be-b972-48bc-b2dc-298d5da131dc-istiod-ca-cert" pod "d59db4be-b972-48bc-b2dc-298d5da131dc" (UID: "d59db4be-b972-48bc-b2dc-298d5da131dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:43.303756 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.303728 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-certs" pod "d59db4be-b972-48bc-b2dc-298d5da131dc" (UID: "d59db4be-b972-48bc-b2dc-298d5da131dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.306424 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.306385 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-podinfo" pod "d59db4be-b972-48bc-b2dc-298d5da131dc" (UID: "d59db4be-b972-48bc-b2dc-298d5da131dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:44:43.307032 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.307004 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-token" pod "d59db4be-b972-48bc-b2dc-298d5da131dc" (UID: "d59db4be-b972-48bc-b2dc-298d5da131dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.307434 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.307408 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-kube-api-access-nbqdp" pod "d59db4be-b972-48bc-b2dc-298d5da131dc" (UID: "d59db4be-b972-48bc-b2dc-298d5da131dc"). InnerVolumeSpecName "kube-api-access-nbqdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.308696 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.308661 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-envoy" pod "d59db4be-b972-48bc-b2dc-298d5da131dc" (UID: "d59db4be-b972-48bc-b2dc-298d5da131dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.404052 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.403974 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-data\") pod \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " Oct 30 18:44:43.404052 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404009 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-socket\") pod \"357860ec-9d4e-4d5e-be19-eec58835b3dc\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " Oct 30 18:44:43.404052 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404031 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istiod-ca-cert\") pod \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404058 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-credential-socket\") pod \"357860ec-9d4e-4d5e-be19-eec58835b3dc\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404084 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-envoy\") pod \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404125 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-credential-socket\") pod \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404197 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-kube-api-access-q9hkg\" (UniqueName: \"kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-kube-api-access-q9hkg\") pod \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404229 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-envoy\") pod \"357860ec-9d4e-4d5e-be19-eec58835b3dc\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404259 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-podinfo\") pod \"357860ec-9d4e-4d5e-be19-eec58835b3dc\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404261 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-socket" pod "357860ec-9d4e-4d5e-be19-eec58835b3dc" (UID: "357860ec-9d4e-4d5e-be19-eec58835b3dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404281 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-data" pod "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" (UID: "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404300 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-certs\") pod \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404356 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357860ec-9d4e-4d5e-be19-eec58835b3dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357860ec-9d4e-4d5e-be19-eec58835b3dc-istiod-ca-cert\") pod \"357860ec-9d4e-4d5e-be19-eec58835b3dc\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404392 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-kube-api-access-whnw9\" (UniqueName: \"kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-kube-api-access-whnw9\") pod \"357860ec-9d4e-4d5e-be19-eec58835b3dc\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404440 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-token\" (UniqueName: \"kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-token\") pod \"357860ec-9d4e-4d5e-be19-eec58835b3dc\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404486 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-data\") pod \"357860ec-9d4e-4d5e-be19-eec58835b3dc\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404513 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-certs\") pod \"357860ec-9d4e-4d5e-be19-eec58835b3dc\" (UID: \"357860ec-9d4e-4d5e-be19-eec58835b3dc\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404547 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-podinfo\") pod \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404573 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-token\" (UniqueName: \"kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-token\") pod \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " Oct 30 18:44:43.404621 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404620 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-socket\") pod \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\" (UID: \"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea\") " Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404988 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.405012 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.405028 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.405042 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.405055 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.405067 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbqdp\" (UniqueName: \"kubernetes.io/projected/d59db4be-b972-48bc-b2dc-298d5da131dc-kube-api-access-nbqdp\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.405084 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d59db4be-b972-48bc-b2dc-298d5da131dc-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.405103 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.405119 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d59db4be-b972-48bc-b2dc-298d5da131dc-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.405133 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.405147 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404390 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-credential-socket" pod "357860ec-9d4e-4d5e-be19-eec58835b3dc" (UID: "357860ec-9d4e-4d5e-be19-eec58835b3dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404481 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istiod-ca-cert" pod "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" (UID: "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404741 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-certs" pod "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" (UID: "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404775 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-credential-socket" pod "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" (UID: "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.404798 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357860ec-9d4e-4d5e-be19-eec58835b3dc-istiod-ca-cert" pod "357860ec-9d4e-4d5e-be19-eec58835b3dc" (UID: "357860ec-9d4e-4d5e-be19-eec58835b3dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.405321 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-certs" pod "357860ec-9d4e-4d5e-be19-eec58835b3dc" (UID: "357860ec-9d4e-4d5e-be19-eec58835b3dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.405749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.405347 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-socket" pod "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" (UID: "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.406792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.406334 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-data" pod "357860ec-9d4e-4d5e-be19-eec58835b3dc" (UID: "357860ec-9d4e-4d5e-be19-eec58835b3dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.406991 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.406961 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-envoy" pod "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" (UID: "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.407102 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.407057 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-envoy" pod "357860ec-9d4e-4d5e-be19-eec58835b3dc" (UID: "357860ec-9d4e-4d5e-be19-eec58835b3dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:43.407300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.407263 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-podinfo" pod "357860ec-9d4e-4d5e-be19-eec58835b3dc" (UID: "357860ec-9d4e-4d5e-be19-eec58835b3dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:44:43.407548 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.407514 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-podinfo" pod "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" (UID: "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:44:43.407757 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.407735 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-token" pod "357860ec-9d4e-4d5e-be19-eec58835b3dc" (UID: "357860ec-9d4e-4d5e-be19-eec58835b3dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.407896 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.407877 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-kube-api-access-whnw9" pod "357860ec-9d4e-4d5e-be19-eec58835b3dc" (UID: "357860ec-9d4e-4d5e-be19-eec58835b3dc"). InnerVolumeSpecName "kube-api-access-whnw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.408260 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.408240 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-kube-api-access-q9hkg" pod "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" (UID: "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea"). InnerVolumeSpecName "kube-api-access-q9hkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.408542 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.408526 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-token" pod "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" (UID: "1c9b738e-f6a3-40d6-b324-26efc6dfe7ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.506251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506215 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506245 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506253 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357860ec-9d4e-4d5e-be19-eec58835b3dc-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506263 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whnw9\" (UniqueName: \"kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-kube-api-access-whnw9\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506274 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506282 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506293 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506304 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506315 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506327 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506334 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506342 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506350 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506358 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506366 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9hkg\" (UniqueName: \"kubernetes.io/projected/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea-kube-api-access-q9hkg\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.506498 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:43.506374 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357860ec-9d4e-4d5e-be19-eec58835b3dc-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:44:44.158581 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.158548 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" Oct 30 18:44:44.158771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.158540 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk" event={"ID":"1c9b738e-f6a3-40d6-b324-26efc6dfe7ea","Type":"ContainerDied","Data":"c9b9f549498fb0fd1b6ebea95a020a16549fdaf72283fceeb9184c6ee9721994"} Oct 30 18:44:44.158771 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.158701 4883 scope.go:122] "RemoveContainer" containerID="45ed41507d39622999d6b7ddbea3fff5d193bf4bdebbf183916bfa2d73f04e8a" Oct 30 18:44:44.160746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.160722 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" Oct 30 18:44:44.160855 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.160742 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5" event={"ID":"d59db4be-b972-48bc-b2dc-298d5da131dc","Type":"ContainerDied","Data":"29049ad536236bb1efbb394340c6e7963e6331bdc99e337f0d7bd53391c92c70"} Oct 30 18:44:44.162222 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.162196 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" Oct 30 18:44:44.162342 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.162199 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r" event={"ID":"357860ec-9d4e-4d5e-be19-eec58835b3dc","Type":"ContainerDied","Data":"3adfc47fa1103f29ce818a0165f08dac3384594c18f1e76e8619ec81b816b157"} Oct 30 18:44:44.170064 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.170038 4883 scope.go:122] "RemoveContainer" containerID="74f73e3788afb15f9fb12fb3b198e70a6a8b6d103eaf4a2637efe694decbd051" Oct 30 18:44:44.179500 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.179481 4883 scope.go:122] "RemoveContainer" containerID="051b3f39ef48e1356b850dc0c578de9b542c6d9c588ae6556a252e8add3edfc3" Oct 30 18:44:44.215424 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.215347 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r"] Oct 30 18:44:44.228734 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.228700 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-zdd4r"] Oct 30 18:44:44.260945 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.260901 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5"] Oct 30 18:44:44.266311 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.266284 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-jhjq5"] Oct 30 18:44:44.306020 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.305988 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk"] Oct 30 18:44:44.311746 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:44.311716 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b424a5-openshift-default-6466f75c99-d46wk"] Oct 30 18:44:45.175906 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:45.175868 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" path="/var/lib/kubelet/pods/1c9b738e-f6a3-40d6-b324-26efc6dfe7ea/volumes" Oct 30 18:44:45.176355 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:45.176340 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" path="/var/lib/kubelet/pods/357860ec-9d4e-4d5e-be19-eec58835b3dc/volumes" Oct 30 18:44:45.176749 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:45.176735 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" path="/var/lib/kubelet/pods/d59db4be-b972-48bc-b2dc-298d5da131dc/volumes" Oct 30 18:44:50.745849 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:50.745813 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-68669df9c4-hb58c_6f0a9864-6eec-4854-bcd3-65d07e04367c/manager/0.log" Oct 30 18:44:50.966788 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:50.966750 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_mcp-gateway-controller-6c98b995c5-5hfqg_244982cc-bbe8-4044-860f-8a766f3562ad/mcp-controller/0.log" Oct 30 18:44:52.618721 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:52.618691 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lw4w4_016abd83-e950-417f-8778-657aa6fe09a6/manager/0.log" Oct 30 18:44:53.233987 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:53.233926 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5456657644-j2hgt_599aec62-8914-4abe-a7e0-3c70148780e6/manager/0.log" Oct 30 18:44:53.782306 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:53.782268 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d66qd_022f46a3-733f-49cc-8190-466a2d40d766/manager/0.log" Oct 30 18:44:54.017576 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:44:54.017544 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xz9f_eee8dbb1-8a5b-4600-893d-78ecd9b403fe/proxy/0.log" Oct 30 18:45:05.076425 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:45:05.076381 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:45:32.040563 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:32.040527 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5"] Oct 30 18:45:32.054079 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:32.054049 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da30caa-2phj5"] Oct 30 18:45:32.884427 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:32.884393 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh"] Oct 30 18:45:32.884620 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:32.884584 4883 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerName="istio-proxy" containerID="cri-o://793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6" gracePeriod=30 Oct 30 18:45:32.948289 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:32.948253 4883 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cxsgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:45:32.948453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:32.948308 4883 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:45:33.042193 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:33.042122 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj"] Oct 30 18:45:33.045007 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:33.044986 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-066e3de6-8klmj"] Oct 30 18:45:33.176775 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:33.176701 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211f0fa1-8896-4b7c-804b-9909d475c6e2" path="/var/lib/kubelet/pods/211f0fa1-8896-4b7c-804b-9909d475c6e2/volumes" Oct 30 18:45:33.177129 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:33.177114 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae7a355-7255-4fcf-b5dd-5bce0659f0df" path="/var/lib/kubelet/pods/5ae7a355-7255-4fcf-b5dd-5bce0659f0df/volumes" Oct 30 18:45:33.689480 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:33.689448 4883 scope.go:122] "RemoveContainer" containerID="43f2b83a93ee874839e7d99cdb33965bc82f9976167dabd65f2706618b9e104f" Oct 30 18:45:33.698148 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:33.698122 4883 scope.go:122] "RemoveContainer" containerID="b77657058c45ef1b11e0c1edaf186e45f5d0aa706ca731414bb378d354b27916" Oct 30 18:45:33.706213 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:33.706194 4883 scope.go:122] "RemoveContainer" containerID="6731998cfd2b553c2fe546d039f8bc827a7ce1ae0108205e27d729292abdbdb0" Oct 30 18:45:35.077714 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:45:35.077679 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:45:38.127104 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.127080 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:45:38.298404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298315 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-credential-socket\") pod \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " Oct 30 18:45:38.298404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298358 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-certs\") pod \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " Oct 30 18:45:38.298404 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298380 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-kube-api-access-tbfqj\" (UniqueName: \"kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-kube-api-access-tbfqj\") pod \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " Oct 30 18:45:38.298739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298414 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-socket\") pod \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " Oct 30 18:45:38.298739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298581 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-envoy\") pod \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " Oct 30 18:45:38.298739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298677 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-data\") pod \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " Oct 30 18:45:38.298739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298666 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-credential-socket" pod "9c1c611e-b4ee-499c-8c86-1409d8461ac7" (UID: "9c1c611e-b4ee-499c-8c86-1409d8461ac7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:38.298739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298704 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-certs" pod "9c1c611e-b4ee-499c-8c86-1409d8461ac7" (UID: "9c1c611e-b4ee-499c-8c86-1409d8461ac7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:38.298739 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298712 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-token\" (UniqueName: \"kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-token\") pod \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " Oct 30 18:45:38.299082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298779 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-socket" pod "9c1c611e-b4ee-499c-8c86-1409d8461ac7" (UID: "9c1c611e-b4ee-499c-8c86-1409d8461ac7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:38.299082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298789 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-podinfo\") pod \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " Oct 30 18:45:38.299082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298857 4883 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istiod-ca-cert\") pod \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\" (UID: \"9c1c611e-b4ee-499c-8c86-1409d8461ac7\") " Oct 30 18:45:38.299082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.298899 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-data" pod "9c1c611e-b4ee-499c-8c86-1409d8461ac7" (UID: "9c1c611e-b4ee-499c-8c86-1409d8461ac7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:38.299300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.299170 4883 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-data\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:45:38.299300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.299191 4883 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-credential-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:45:38.299300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.299207 4883 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-certs\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:45:38.299300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.299221 4883 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-workload-socket\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:45:38.299300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.299270 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istiod-ca-cert" pod "9c1c611e-b4ee-499c-8c86-1409d8461ac7" (UID: "9c1c611e-b4ee-499c-8c86-1409d8461ac7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:38.300844 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.300811 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-envoy" pod "9c1c611e-b4ee-499c-8c86-1409d8461ac7" (UID: "9c1c611e-b4ee-499c-8c86-1409d8461ac7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:38.301006 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.300904 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-kube-api-access-tbfqj" pod "9c1c611e-b4ee-499c-8c86-1409d8461ac7" (UID: "9c1c611e-b4ee-499c-8c86-1409d8461ac7"). InnerVolumeSpecName "kube-api-access-tbfqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:38.301186 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.301158 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-podinfo" pod "9c1c611e-b4ee-499c-8c86-1409d8461ac7" (UID: "9c1c611e-b4ee-499c-8c86-1409d8461ac7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:45:38.301272 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.301182 4883 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-token" pod "9c1c611e-b4ee-499c-8c86-1409d8461ac7" (UID: "9c1c611e-b4ee-499c-8c86-1409d8461ac7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:38.399681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.399645 4883 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-token\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:45:38.399681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.399673 4883 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-podinfo\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:45:38.399681 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.399686 4883 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istiod-ca-cert\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:45:38.399915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.399699 4883 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbfqj\" (UniqueName: \"kubernetes.io/projected/9c1c611e-b4ee-499c-8c86-1409d8461ac7-kube-api-access-tbfqj\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:45:38.399915 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.399712 4883 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1c611e-b4ee-499c-8c86-1409d8461ac7-istio-envoy\") on node \"ip-10-0-130-250.ec2.internal\" DevicePath \"\"" Oct 30 18:45:38.443169 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.443132 4883 generic.go:356] "Generic (PLEG): container finished" podID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerID="793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6" exitCode=0 Oct 30 18:45:38.443300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.443178 4883 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" Oct 30 18:45:38.443300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.443190 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" event={"ID":"9c1c611e-b4ee-499c-8c86-1409d8461ac7","Type":"ContainerDied","Data":"793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6"} Oct 30 18:45:38.443300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.443238 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh" event={"ID":"9c1c611e-b4ee-499c-8c86-1409d8461ac7","Type":"ContainerDied","Data":"c7c9f9ecc8f296c89b6e9b362d0841932228ed1cbae20a90aebcf0f8cf1d178c"} Oct 30 18:45:38.443300 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.443260 4883 scope.go:122] "RemoveContainer" containerID="793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6" Oct 30 18:45:38.452545 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.452527 4883 scope.go:122] "RemoveContainer" containerID="793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6" Oct 30 18:45:38.452807 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:45:38.452785 4883 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6\": container with ID starting with 793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6 not found: ID does not exist" containerID="793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6" Oct 30 18:45:38.452910 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.452812 4883 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6"} err="failed to get container status \"793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6\": rpc error: code = NotFound desc = could not find container \"793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6\": container with ID starting with 793a1b75c20d6ad15756c96c1449f644828f5dd783e5e5c9bb075accc6064ae6 not found: ID does not exist" Oct 30 18:45:38.498817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.498782 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh"] Oct 30 18:45:38.512134 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:38.512104 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cxsgh"] Oct 30 18:45:39.039985 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:39.039949 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd"] Oct 30 18:45:39.046042 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:39.046019 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r"] Oct 30 18:45:39.053886 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:39.053853 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f3207b-hdnbd"] Oct 30 18:45:39.054843 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:39.054820 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c26a381e-r4b4r"] Oct 30 18:45:39.177188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:39.177158 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60578dc3-222d-4689-8103-e0782b2de3d1" path="/var/lib/kubelet/pods/60578dc3-222d-4689-8103-e0782b2de3d1/volumes" Oct 30 18:45:39.177585 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:39.177572 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" path="/var/lib/kubelet/pods/9c1c611e-b4ee-499c-8c86-1409d8461ac7/volumes" Oct 30 18:45:39.177927 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:39.177915 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d394ecb7-4864-4387-88cb-a07bf6e39af1" path="/var/lib/kubelet/pods/d394ecb7-4864-4387-88cb-a07bf6e39af1/volumes" Oct 30 18:45:46.046561 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:46.046529 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lw4w4_016abd83-e950-417f-8778-657aa6fe09a6/manager/0.log" Oct 30 18:45:46.621239 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:46.621208 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5456657644-j2hgt_599aec62-8914-4abe-a7e0-3c70148780e6/manager/0.log" Oct 30 18:45:47.008626 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:47.008523 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d66qd_022f46a3-733f-49cc-8190-466a2d40d766/manager/0.log" Oct 30 18:45:47.219075 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:47.219045 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xz9f_eee8dbb1-8a5b-4600-893d-78ecd9b403fe/proxy/0.log" Oct 30 18:45:48.242514 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:48.242480 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6fd00bab-t6kl5_f3342acd-bd0f-4e2e-b346-130a82fc2589/revoke-keys/0.log" Oct 30 18:45:48.435865 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:45:48.435837 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-88939678-hkrfm_70393036-41e6-4ede-92b9-59eb457bb4c5/revoke-keys/0.log" Oct 30 18:46:00.833799 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:00.833758 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_developer-portal-controller-66d8dbb8c8-d7cds_282b0162-4f29-4198-a4dc-50b21b6266b6/manager/0.log" Oct 30 18:46:02.008355 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:02.008321 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-68669df9c4-hb58c_6f0a9864-6eec-4854-bcd3-65d07e04367c/manager/0.log" Oct 30 18:46:02.222956 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:02.222902 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_mcp-gateway-controller-6c98b995c5-5hfqg_244982cc-bbe8-4044-860f-8a766f3562ad/mcp-controller/0.log" Oct 30 18:46:04.014082 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:04.014044 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c55b8cc4-f2d66_5e6711b6-3b88-438e-958e-f64432be2ddc/kube-auth-proxy/0.log" Oct 30 18:46:05.077750 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:46:05.077712 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:46:05.213851 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:05.213817 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f6d947f75-ln5z8_4260cb2f-efee-4b45-9f73-a48757f3ce75/router/0.log" Oct 30 18:46:17.034819 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:17.034776 4883 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm"] Oct 30 18:46:17.048088 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:17.048059 4883 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88939678-hkrfm"] Oct 30 18:46:17.177759 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:17.177725 4883 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70393036-41e6-4ede-92b9-59eb457bb4c5" path="/var/lib/kubelet/pods/70393036-41e6-4ede-92b9-59eb457bb4c5/volumes" Oct 30 18:46:33.743402 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:33.743368 4883 scope.go:122] "RemoveContainer" containerID="5b591d631795c86584ea55373f22cc1df44689810642fe025ff2df716ac10a2c" Oct 30 18:46:33.751777 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:33.751757 4883 scope.go:122] "RemoveContainer" containerID="8f14abc3c0ec58bb095a90334454460c0ca8d5316922be0ef96a5fb28b9d41a4" Oct 30 18:46:33.759259 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:33.759242 4883 scope.go:122] "RemoveContainer" containerID="6bdaeedbba98d19ccb1c7b48773c72f2447bc520928d5670f321453a5e9e8a5a" Oct 30 18:46:33.766630 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:33.766606 4883 scope.go:122] "RemoveContainer" containerID="f01d08236981a8eb81de552c08e4f8f65acd25bea04874ea21a55b2a07eab0e2" Oct 30 18:46:33.773714 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:33.773698 4883 scope.go:122] "RemoveContainer" containerID="fd2a9a7d7bf50c2477df1cc6aa50180e751c0d62eeff1e79fe42ba84e8ef7dc8" Oct 30 18:46:33.780970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:33.780950 4883 scope.go:122] "RemoveContainer" containerID="f29cb08f4912f898b4beeee3f022830ea00682ea88685b3714cf452911aa0e97" Oct 30 18:46:34.010244 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:34.010172 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fglxs_44ce8b1c-6ab2-4f83-a1f6-183cc6cea104/global-pull-secret-syncer/0.log" Oct 30 18:46:34.373801 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:34.373777 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-2tjnd_b6956a31-29a1-4cd7-bc8b-9cda43dc2e73/connection-checker/0.log" Oct 30 18:46:34.601605 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:34.601577 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wjjnl_06498a3b-c75b-4bf5-9fc7-beced86dbfc5/connection-checker/0.log" Oct 30 18:46:35.005286 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:35.005257 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vdlbp_2ce04018-ba31-464c-8996-967b9100f174/konnectivity-agent/0.log" Oct 30 18:46:35.077841 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:46:35.077811 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:46:35.264531 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:35.264449 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-250.ec2.internal_ef96f59234cd19c8b7af6a47596c7964/haproxy/0.log" Oct 30 18:46:42.205364 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:42.205331 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_developer-portal-controller-66d8dbb8c8-d7cds_282b0162-4f29-4198-a4dc-50b21b6266b6/manager/0.log" Oct 30 18:46:43.682574 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:43.682542 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-68669df9c4-hb58c_6f0a9864-6eec-4854-bcd3-65d07e04367c/manager/0.log" Oct 30 18:46:43.774085 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:43.774058 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_mcp-gateway-controller-6c98b995c5-5hfqg_244982cc-bbe8-4044-860f-8a766f3562ad/mcp-controller/0.log" Oct 30 18:46:45.512231 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512192 4883 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg"] Oct 30 18:46:45.512835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512755 4883 cpu_manager.go:414] "containerMap: removing container" podUID="f3342acd-bd0f-4e2e-b346-130a82fc2589" containerName="revoke-keys" Oct 30 18:46:45.512835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512775 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3342acd-bd0f-4e2e-b346-130a82fc2589" containerName="revoke-keys" Oct 30 18:46:45.512835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512804 4883 cpu_manager.go:414] "containerMap: removing container" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" Oct 30 18:46:45.512835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512813 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" Oct 30 18:46:45.512835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512828 4883 cpu_manager.go:414] "containerMap: removing container" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" Oct 30 18:46:45.512835 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512835 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" Oct 30 18:46:45.513217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512863 4883 cpu_manager.go:414] "containerMap: removing container" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" Oct 30 18:46:45.513217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512872 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" Oct 30 18:46:45.513217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512903 4883 cpu_manager.go:414] "containerMap: removing container" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerName="istio-proxy" Oct 30 18:46:45.513217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512912 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerName="istio-proxy" Oct 30 18:46:45.513217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512924 4883 cpu_manager.go:414] "containerMap: removing container" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" Oct 30 18:46:45.513217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.512958 4883 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" Oct 30 18:46:45.513217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.513059 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1c611e-b4ee-499c-8c86-1409d8461ac7" containerName="istio-proxy" Oct 30 18:46:45.513217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.513070 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9b738e-f6a3-40d6-b324-26efc6dfe7ea" containerName="istio-proxy" Oct 30 18:46:45.513217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.513083 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3342acd-bd0f-4e2e-b346-130a82fc2589" containerName="revoke-keys" Oct 30 18:46:45.513217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.513096 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="357860ec-9d4e-4d5e-be19-eec58835b3dc" containerName="istio-proxy" Oct 30 18:46:45.513217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.513106 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff654c6-de28-43a9-a216-6dc69a9a6752" containerName="main" Oct 30 18:46:45.513217 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.513118 4883 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59db4be-b972-48bc-b2dc-298d5da131dc" containerName="istio-proxy" Oct 30 18:46:45.517597 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.517576 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.537645 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.537623 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg"] Oct 30 18:46:45.549188 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.549169 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lwz4v\"/\"kube-root-ca.crt\"" Oct 30 18:46:45.549309 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.549237 4883 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lwz4v\"/\"openshift-service-ca.crt\"" Oct 30 18:46:45.549309 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.549238 4883 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lwz4v\"/\"default-dockercfg-9zdsh\"" Oct 30 18:46:45.584435 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.584408 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-podres\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.584546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.584445 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5nvl\" (UniqueName: \"kubernetes.io/projected/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-kube-api-access-h5nvl\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.584546 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.584465 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-sys\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.584639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.584586 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-lib-modules\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.584639 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.584622 4883 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-proc\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.685297 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.685261 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-lib-modules\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.685453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.685308 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-proc\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.685453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.685354 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-podres\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.685453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.685378 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5nvl\" (UniqueName: \"kubernetes.io/projected/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-kube-api-access-h5nvl\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.685453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.685399 4883 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-sys\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.685453 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.685405 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-lib-modules\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.685623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.685476 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-podres\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.685623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.685492 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-sys\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.685623 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.685515 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-proc\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.710111 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.710073 4883 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5nvl\" (UniqueName: \"kubernetes.io/projected/c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed-kube-api-access-h5nvl\") pod \"perf-node-gather-daemonset-vcpmg\" (UID: \"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed\") " pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:45.829970 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:45.829878 4883 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:46.133484 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:46.133450 4883 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg"] Oct 30 18:46:46.136542 ip-10-0-130-250 kubenswrapper[4883]: W1030 18:46:46.136512 4883 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc2c9fc9b_30fe_4c4b_86c4_6844a1ba93ed.slice/crio-88c2b0eda0defdc89b63877c5ae219987f86aa82c7a0f8aabc7958bcba68cc6e WatchSource:0}: Error finding container 88c2b0eda0defdc89b63877c5ae219987f86aa82c7a0f8aabc7958bcba68cc6e: Status 404 returned error can't find the container with id 88c2b0eda0defdc89b63877c5ae219987f86aa82c7a0f8aabc7958bcba68cc6e Oct 30 18:46:46.325171 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:46.325137 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-z2t26_867cb674-bd6b-4ca7-98a6-fa31bdaa132d/cluster-monitoring-operator/0.log" Oct 30 18:46:46.797898 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:46.797864 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" event={"ID":"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed","Type":"ContainerStarted","Data":"b4990147ec56b5ce5e6fbd3825a0f14160d923bbc45cab5cef236216c0344717"} Oct 30 18:46:46.798318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:46.797914 4883 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" event={"ID":"c2c9fc9b-30fe-4c4b-86c4-6844a1ba93ed","Type":"ContainerStarted","Data":"88c2b0eda0defdc89b63877c5ae219987f86aa82c7a0f8aabc7958bcba68cc6e"} Oct 30 18:46:46.798318 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:46.797953 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:46.842069 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:46.842019 4883 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" podStartSLOduration=1.842000061 podStartE2EDuration="1.842000061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:46.840803715 +0000 UTC m=+3265.797807551" watchObservedRunningTime="2026-10-30 18:46:46.842000061 +0000 UTC m=+3265.799003898" Oct 30 18:46:47.457482 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:47.457447 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8jm7_bc2ac4e0-5b2c-4e3d-bffa-def481c62498/node-exporter/0.log" Oct 30 18:46:47.567551 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:47.567503 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8jm7_bc2ac4e0-5b2c-4e3d-bffa-def481c62498/kube-rbac-proxy/0.log" Oct 30 18:46:47.686869 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:47.686826 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8jm7_bc2ac4e0-5b2c-4e3d-bffa-def481c62498/init-textfile/0.log" Oct 30 18:46:48.261466 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:48.261432 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lw4w4_016abd83-e950-417f-8778-657aa6fe09a6/manager/0.log" Oct 30 18:46:48.416202 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:48.416158 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lw4w4_016abd83-e950-417f-8778-657aa6fe09a6/manager/0.log" Oct 30 18:46:48.773725 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:48.773695 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72c5389e-4e5d-49a0-9dea-8b460821f353/prometheus/0.log" Oct 30 18:46:48.794917 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:48.794885 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5456657644-j2hgt_599aec62-8914-4abe-a7e0-3c70148780e6/manager/0.log" Oct 30 18:46:48.898558 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:48.898523 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72c5389e-4e5d-49a0-9dea-8b460821f353/config-reloader/0.log" Oct 30 18:46:48.905096 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:48.905066 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5456657644-j2hgt_599aec62-8914-4abe-a7e0-3c70148780e6/manager/0.log" Oct 30 18:46:49.031446 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:49.031376 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72c5389e-4e5d-49a0-9dea-8b460821f353/thanos-sidecar/0.log" Oct 30 18:46:49.267708 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:49.267676 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72c5389e-4e5d-49a0-9dea-8b460821f353/kube-rbac-proxy-web/0.log" Oct 30 18:46:49.570916 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:49.570881 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d66qd_022f46a3-733f-49cc-8190-466a2d40d766/manager/0.log" Oct 30 18:46:49.687407 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:49.687375 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d66qd_022f46a3-733f-49cc-8190-466a2d40d766/manager/0.log" Oct 30 18:46:49.775233 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:49.775200 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xz9f_eee8dbb1-8a5b-4600-893d-78ecd9b403fe/proxy/0.log" Oct 30 18:46:49.778543 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:49.778521 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72c5389e-4e5d-49a0-9dea-8b460821f353/kube-rbac-proxy/0.log" Oct 30 18:46:49.842167 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:49.842144 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xz9f_eee8dbb1-8a5b-4600-893d-78ecd9b403fe/proxy/0.log" Oct 30 18:46:49.891448 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:49.891411 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72c5389e-4e5d-49a0-9dea-8b460821f353/kube-rbac-proxy-thanos/0.log" Oct 30 18:46:50.002786 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:50.002760 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_72c5389e-4e5d-49a0-9dea-8b460821f353/init-config-reloader/0.log" Oct 30 18:46:52.707495 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:52.707456 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lw4w4_016abd83-e950-417f-8778-657aa6fe09a6/manager/0.log" Oct 30 18:46:52.815329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:52.815301 4883 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lwz4v/perf-node-gather-daemonset-vcpmg" Oct 30 18:46:52.820295 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:52.820253 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lw4w4_016abd83-e950-417f-8778-657aa6fe09a6/manager/0.log" Oct 30 18:46:53.256104 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:53.256063 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5456657644-j2hgt_599aec62-8914-4abe-a7e0-3c70148780e6/manager/0.log" Oct 30 18:46:53.343066 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:53.343027 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5456657644-j2hgt_599aec62-8914-4abe-a7e0-3c70148780e6/manager/0.log" Oct 30 18:46:54.104292 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:54.104240 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d66qd_022f46a3-733f-49cc-8190-466a2d40d766/manager/0.log" Oct 30 18:46:54.167480 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:54.167445 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xz9f_eee8dbb1-8a5b-4600-893d-78ecd9b403fe/proxy/0.log" Oct 30 18:46:54.213354 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:54.213325 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d66qd_022f46a3-733f-49cc-8190-466a2d40d766/manager/0.log" Oct 30 18:46:54.275678 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:54.275641 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/2.log" Oct 30 18:46:54.282414 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:54.282386 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5zcf9_4fb42c05-fb07-4dae-975b-71d45dd52e03/console-operator/3.log" Oct 30 18:46:54.291374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:54.291354 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xz9f_eee8dbb1-8a5b-4600-893d-78ecd9b403fe/proxy/0.log" Oct 30 18:46:57.039338 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:57.039302 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-hpv5m_ba65a282-0992-49ce-b4bc-108c8f69016d/volume-data-source-validator/0.log" Oct 30 18:46:57.144063 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:57.144028 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lw4w4_016abd83-e950-417f-8778-657aa6fe09a6/manager/0.log" Oct 30 18:46:57.642609 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:57.642572 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lw4w4_016abd83-e950-417f-8778-657aa6fe09a6/manager/0.log" Oct 30 18:46:57.708374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:57.708334 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5456657644-j2hgt_599aec62-8914-4abe-a7e0-3c70148780e6/manager/0.log" Oct 30 18:46:58.158398 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:58.158360 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5456657644-j2hgt_599aec62-8914-4abe-a7e0-3c70148780e6/manager/0.log" Oct 30 18:46:58.575301 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:58.575223 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d66qd_022f46a3-733f-49cc-8190-466a2d40d766/manager/0.log" Oct 30 18:46:58.637145 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:58.637104 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xz9f_eee8dbb1-8a5b-4600-893d-78ecd9b403fe/proxy/0.log" Oct 30 18:46:58.930792 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:58.930744 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-d66qd_022f46a3-733f-49cc-8190-466a2d40d766/manager/0.log" Oct 30 18:46:58.985254 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:58.985223 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xz9f_eee8dbb1-8a5b-4600-893d-78ecd9b403fe/proxy/0.log" Oct 30 18:46:59.843400 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:59.843360 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6fncr_7cf0aaaf-7ca8-44b7-afad-829e4455e56a/dns/0.log" Oct 30 18:46:59.955800 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:46:59.955738 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6fncr_7cf0aaaf-7ca8-44b7-afad-829e4455e56a/kube-rbac-proxy/0.log" Oct 30 18:47:00.819866 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:00.819834 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xbxvk_97c8fb7c-e9a6-4eff-9a66-b70848c03a95/dns-node-resolver/0.log" Oct 30 18:47:02.317298 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:02.317269 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7gt8q_d3ebd2d3-bb3b-4d9e-a95c-027b7ad0bdd0/node-ca/0.log" Oct 30 18:47:03.397590 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:03.397483 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c55b8cc4-f2d66_5e6711b6-3b88-438e-958e-f64432be2ddc/kube-auth-proxy/0.log" Oct 30 18:47:03.829554 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:03.829480 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f6d947f75-ln5z8_4260cb2f-efee-4b45-9f73-a48757f3ce75/router/0.log" Oct 30 18:47:04.937169 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:04.937139 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c55b8cc4-f2d66_5e6711b6-3b88-438e-958e-f64432be2ddc/kube-auth-proxy/0.log" Oct 30 18:47:05.082057 ip-10-0-130-250 kubenswrapper[4883]: E1030 18:47:05.082025 4883 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sw2jb" containerName="extractor" Oct 30 18:47:05.263162 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:05.263063 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c55b8cc4-f2d66_5e6711b6-3b88-438e-958e-f64432be2ddc/kube-auth-proxy/0.log" Oct 30 18:47:05.330571 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:05.330541 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f6d947f75-ln5z8_4260cb2f-efee-4b45-9f73-a48757f3ce75/router/0.log" Oct 30 18:47:05.699430 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:05.699397 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f6d947f75-ln5z8_4260cb2f-efee-4b45-9f73-a48757f3ce75/router/0.log" Oct 30 18:47:07.408374 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:07.408340 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c55b8cc4-f2d66_5e6711b6-3b88-438e-958e-f64432be2ddc/kube-auth-proxy/0.log" Oct 30 18:47:08.127697 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:08.127668 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f6d947f75-ln5z8_4260cb2f-efee-4b45-9f73-a48757f3ce75/router/0.log" Oct 30 18:47:09.872251 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:09.872221 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_developer-portal-controller-66d8dbb8c8-d7cds_282b0162-4f29-4198-a4dc-50b21b6266b6/manager/0.log" Oct 30 18:47:10.265823 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:10.265738 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-68669df9c4-hb58c_6f0a9864-6eec-4854-bcd3-65d07e04367c/manager/0.log" Oct 30 18:47:10.299152 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:10.299116 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_mcp-gateway-controller-6c98b995c5-5hfqg_244982cc-bbe8-4044-860f-8a766f3562ad/mcp-controller/0.log" Oct 30 18:47:10.464516 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:10.464468 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_developer-portal-controller-66d8dbb8c8-d7cds_282b0162-4f29-4198-a4dc-50b21b6266b6/manager/0.log" Oct 30 18:47:10.848247 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:10.848206 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-68669df9c4-hb58c_6f0a9864-6eec-4854-bcd3-65d07e04367c/manager/0.log" Oct 30 18:47:10.877107 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:10.877073 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c55b8cc4-f2d66_5e6711b6-3b88-438e-958e-f64432be2ddc/kube-auth-proxy/0.log" Oct 30 18:47:10.880305 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:10.880282 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_mcp-gateway-controller-6c98b995c5-5hfqg_244982cc-bbe8-4044-860f-8a766f3562ad/mcp-controller/0.log" Oct 30 18:47:11.061643 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:11.061613 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t5n25_c1aa6b58-d1c3-40a0-873c-86b44c438fac/serve-healthcheck-canary/0.log" Oct 30 18:47:11.268817 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:11.268729 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f6d947f75-ln5z8_4260cb2f-efee-4b45-9f73-a48757f3ce75/router/0.log" Oct 30 18:47:12.250329 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:12.250286 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-csr4p_15f0fc8a-91c4-404a-91ec-b39914e83cae/insights-operator/0.log" Oct 30 18:47:12.252315 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:12.252295 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-csr4p_15f0fc8a-91c4-404a-91ec-b39914e83cae/insights-operator/1.log" Oct 30 18:47:13.053924 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:13.053895 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sw2jb_9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa/kube-rbac-proxy/0.log" Oct 30 18:47:13.162485 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:13.162458 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sw2jb_9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa/exporter/0.log" Oct 30 18:47:13.275262 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:13.275233 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sw2jb_9a7f9d5c-8fbe-439e-8ab5-3eb7891b01aa/extractor/0.log" Oct 30 18:47:13.404302 ip-10-0-130-250 kubenswrapper[4883]: I1030 18:47:13.404271 4883 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-sv7x6-6h4f9_b5afb42c-f785-4b8f-a6ca-2236d724d8af/insights-gathering/0.log"